At the recent Embedded Vision Summit, Chen Wu, Head of Perception at Waymo, gave an insightful presentation titled “Building Trustworthy Autonomous Driving Systems: Waymo’s Holistic AI Safety Approach.” The session offered a rare look into how one of the industry leaders scales autonomous driving technology from a pure engineering standpoint.
For those of us focused on performance engineering and embedded AI optimization, the data presented was highly relevant. Waymo is now running a full-scale public service, facilitating over 500,000 paid rides per week across 11 major U.S. cities, with active testing expanding to London and Tokyo. Their numbers show an 92% reduction in serious injury crashes compared to human drivers over the same mileage. Achieving this level of reliability requires a tight integration between advanced AI models and the physical hardware on the vehicle—a core challenge we tackle every day.
Handling the Edge Cases of Multi-Modal Sensor Fusion
During her talk, Wu demonstrated how the “Waymo Driver” navigates complex, chaotic situations, such as a motorcyclist falling directly in front of the vehicle or pedestrians appearing from unexpected blind spots at night.
A particularly striking example was a severe dust storm in Phoenix. While the vehicle’s high-definition cameras saw nothing but an opaque yellow fog, the onboard LiDAR and Radar sensors perfectly tracked a pedestrian standing on the side of the road. Waymo relies heavily on this 360-degree multi-modal sensor fusion to ensure zero blind spots under any weather condition.
From a performance engineering perspective, integrating data from LiDAR, Radar, cameras, and microphones in real time creates a massive computational burden. Furthermore, autonomous driving stacks are rapidly shifting from traditional CNNs to complex Transformers and large Vision-Language Models (VLMs) to handle sophisticated semantic reasoning.
Running these multi-layered workloads natively on a vehicle under strict latency, power, and thermal limits is incredibly difficult. This is exactly where our work at Fixstars comes into play. Through dedicated AI model porting, quantization, and hardware-specific optimization, we resolve these processing bottlenecks so that vehicles can analyze critical environmental changes in milliseconds without overloading the hardware.
Onboard Deterministic Control and Hardware Optimization
Another core theme of Waymo’s architecture is its component-based safety approach. Instead of relying on an end-to-end “black box” model, Waymo breaks the driving pipeline down into distinct components—perception, prediction, and planning—all backed by a deterministic safety guardrail layer.
This architecture was further explained during the Q&A when an audience member asked if human operators remotely steer the vehicles during tricky edge cases. Wu clarified that remote steering is not used because network latency makes it inherently unsafe.
Instead, Waymo uses a system called Remote Assistance. The onboard AI maintains absolute control over steering and braking, but it can query a remote operator for high-level guidance (e.g., “Is this lane permanently closed due to construction?”). The human provides a simple confirmation, and the car’s local system safely executes the path.
This reliance on local, deterministic processing highlights a major engineering reality: safety-critical edge applications cannot depend on the cloud or tolerate sluggish execution. Every component must be optimized to run efficiently on the vehicle’s specific chip architecture.
At Fixstars, we have spent years bridging the gap between cutting-edge software and complex automotive silicon. For example, our experience includes co-developing optimization toolsets for the automotive R-Car SoC, enabling advanced ADAS and autonomous features to run at peak efficiency. Whether optimization is targeted at GPUs, NPUs, or FPGAs, ensuring that a model behaves predictably and fast on production-grade chips is essential to moving autonomous driving from a prototype to a commercial reality.
Learn more about Fixstars’ optimization services
Conclusion
The evolution of autonomous driving AI is moving faster than ever. As models expand to handle unpredictable human behaviors, the underlying software must keep pace. A driving algorithm is only as good as its execution speed; it has to process safety logic before a collision becomes unavoidable.
By continuously refining our performance engineering methodologies, we help our partners push their embedded models to the hardware limit, ensuring that next-generation mobility remains both smart and safe.
Author
