At the recent Embedded Vision Summit, Pullarao Maddu, Embedded AI Architect at Valeo Vision Systems, delivered a highly practical presentation titled “Decomposing AI Pipelines for Robust Performance at the Edge.” For systems and performance engineers, the session went straight to the core of a persistent deployment challenge: why a model that looks perfect on a desktop evaluation frequently encounters performance hurdles when deployed onto production-grade edge silicon.
Maddu walked through Valeo’s transition from Gen 1 CNN pipelines (running on chips like the Renesas V3H1 at 0.64 TOPS) to next-generation Bird’s Eye View (BEV) multi-task pipelines running on newer platforms like the Qualcomm SA8650 at 9.8 TOPS.
As a performance engineering company that focuses daily on optimizing embedded AI and mapping complex software to target hardware limits, we found his technical breakdown of system-level constraints incredibly aligned with our own development experiences. Below are our key engineering takeaways from the session and how they connect to real-world software acceleration.
1. The Real Enemy: Latency Jitter, Not Average Performance
In safety-critical embedded systems like automotive ADAS, marketing metrics like “Average FPS” or “Peak TOPS” are often misleading. Maddu emphasized that determinism is an end-to-end systems property involving the model architecture, compiler, runtime, OS, memory subsystem, and I/O.
When a pipeline experiences variable execution times, subsequent frames suffer from scheduling “bleed,” which can push processing past the critical hard deadline. The real enemy of edge deployment is latency jitter—the variance caused by data traffic, hardware core contention, Quality of Service (QoS) degradation, and pipeline composition.
You cannot optimize inference in isolation. Bounded latency must be intentionally architected across the entire system, ensuring deadline compliance rather than aiming for brief bursts of peak theoretical performance.
2. Why FLOPs and MACs Do Not Predict Real Latency
A common pitfall in AI development is assuming that two models with the exact same Multiply-Accumulate (MAC) count or operational footprint will execute with the same latency. In reality, edge hardware compute capabilities have scaled significantly faster than external memory bandwidth, driving up compute-to-memory ratios.
If a model’s operational intensity places it in a bandwidth-bound region, adding more processing units or choosing a chip with higher peak TOPS will not reduce latency.
Before rewriting custom compute kernels, engineers must conduct a thorough roofline-style analysis. This means auditing the precise bytes transferred per frame and counting the DRAM round-trips. Minimizing memory access and structuring data layout to maximize cache reuse is frequently far more impactful than trying to optimize raw mathematical operations.
3. Heterogeneous SoC Orchestration and the “Boundary Cost”
Modern edge processors are deeply heterogeneous, pack-integrating various combinations of CPUs, GPUs, NPUs, and DSPs on a single SoC. While distributing tasks across these blocks is necessary (e.g., leaving sequential post-processing to the CPU while pushing dense parallel backbones to the NPU), this division of labor introduces significant system overhead.
[Camera/Sensors] ➔ [ISP Processing] ➔ [NPU Inference] ➔ [CPU Post-Process]
│ │ │ │
└─────────── Boundary Costs & Synchronization Fences ────┘
Every transition between different compute cores introduces submission overhead, synchronization fences, and cache invalidation traffic. Furthermore, the chip’s internal interconnect topology and cache coherency model decide whether you can achieve true “zero-copy” data transfers. Non-coherent accelerators require explicit software cache maintenance and fencing around DMA operations, which can heavily impact tail latency.
At Fixstars, when we port complex multi-sensor pipelines to platforms like the Renesas R-Car SoC or modern heterogeneous hardware, our primary target is eliminating these hidden boundary costs. Leaving data routing purely to generic compilers often results in copy amplification. Efficient orchestration requires hand-budgeting memory fences and writing clean, pipelined data streams that allow compute blocks to process data concurrently rather than waiting on sequential sync locks.
4. Breaking Through the “Software Wall” and Operator Gaps
Even if the hardware architecture is perfectly balanced, development teams frequently hit a “software wall” during model compilation and deployment. Maddu highlighted two primary issues here:
- The Operator Gap: An optimized model graph is only as fast as its weakest kernel. Runtimes on NPUs and DSPs often lack native support for memory-intensive data operators such as
Gather,Scatter, orCumsum. When a compiler encounters these unsupported layers, it forces a fallback to the CPU, stalling the accelerator pipeline. - The Dimension Trap: Many deployment runtimes degrade in performance or choke entirely when handling dynamic shapes or high-dimensional (6D+) tensors. Edge hardware relies heavily on flat, fixed memory addresses rather than flexible mathematical abstractions.
When exporting a model through common formats like ONNX, it is easy to end up with a fragmented “Frankenstein graph” full of reshape and transpose layers that degrade execution efficiency.
Resolving this requires deep compilation insight. When we encounter these operator gaps in client projects, our approach is to bypass general compiler fallbacks by writing highly optimized, custom low-level kernels directly targeting the NPU or DSP instruction set. By flattening tensor dimensions pre-export and substituting complex operations with hardware-friendly equivalents, we ensure the graph remains entirely within the fast path of the accelerator.
Conclusion
As Valeo’s pipeline analysis illustrated, actual raw model inference often accounts for just a fraction of the total glass-to-glass pipeline latency (e.g., less than 15ms out of a total 134ms pipeline loop). The remaining time is entirely consumed by image acquisition, ISP processing, color conversion, queuing, and post-processing.
Optimizing the neural network alone will not yield a production-ready system. Real-world deployment requires a holistic performance engineering approach that profiles and refines the entire data path from the lens sensor to the final actuation command. By treating hardware topology, memory bandwidth, and kernel orchestration as tightly coupled constraints, developers can reliably move advanced vision pipelines off the laboratory testbed and scale them confidently across large production fleets.
Author
