Autonomous Vehicles
Autonomous vehicles are the most heavily-funded application of physical AI, with over $100 billion invested across the industry since 2015. They are also the most demanding: millisecond decision-making, life-critical safety requirements, and operation in an infinitely varied real-world environment. Understanding AV technology provides the clearest window into the engineering challenges of physical AI at scale.
SAE Levels of Automation
| Level | Name | Description |
|---|---|---|
| L0 | No automation | Human controls everything |
| L1 | Driver assistance | Cruise control, lane keep assist - one function at a time |
| L2 | Partial automation | Multiple functions (Tesla Autopilot, GM Super Cruise) - human must monitor |
| L3 | Conditional automation | System drives in defined conditions; human must be ready to take over |
| L4 | High automation | No human needed in defined area (Waymo robotaxi - human optional) |
| L5 | Full automation | Anywhere, any conditions - not yet achieved commercially |
The AV Perception Stack
An AV must understand its complete 360° environment in real time. Two competing approaches:
Sensor-fusion (Waymo approach):
Cameras (360°) ─┐
├──▶ Sensor Fusion ──▶ 3D Scene Model
LIDAR ─┤ (deep learning) (objects, lanes,
│ free space)
RADAR ─┘- LIDAR provides precise 3D geometry regardless of lighting
- Cameras provide rich semantic information (traffic lights, signs, pedestrian pose)
- RADAR provides velocity information through fog and rain
- Sensor fusion combines all three for redundancy and accuracy
Vision-only (Tesla approach):
8 Cameras (360°) ──▶ Neural network ──▶ Occupancy grid
(end-to-end, + trajectory
trained on prediction
millions of
human miles)- No LIDAR - pure camera input, like humans drive
- End-to-end neural network trained on fleet data from millions of vehicles
- Tesla's "FSD v12" is a single large transformer - inputs are camera frames, outputs are steering, acceleration, braking
- Advantage: cheaper hardware; disadvantage: harder to guarantee safety in edge cases
Waymo - The L4 Leader
Waymo (Alphabet) operates the only commercial L4 robotaxi service in the US. Key facts as of 2025–2026:
- Operating in Phoenix, San Francisco, and Los Angeles - fully driverless in defined geo-fenced areas
- Over 50,000 trips per week, 1 million+ robotaxi miles without a serious at-fault crash
- Waymo One app - book a robotaxi, no safety driver present
- Custom sensor suite: cameras + 5th-generation Lidar + radar + proprietary compute
- High-definition maps of every inch of operational area - the AV knows every lane line, stop sign, and speed bump before driving
- Limitation: expansion to new cities requires expensive mapping and regulatory approval; cannot drive unmapped roads
Tesla FSD - The Fleet Data Approach
Tesla's Full Self-Driving (FSD) v12 represents a philosophical shift from rule-based systems to learned end-to-end driving:
- Single neural network: 8 camera streams in → steering/acceleration/braking out
- Trained on "intervention" data - when a human driver takes over, that moment is labelled and used for training
- Fleet of ~5 million Tesla vehicles generates training data continuously
- Current status: FSD requires driver supervision (L2/L3) - not L4
- Tesla's bet: at sufficient scale of data and model size, FSD will generalize to L4+ without mapping every road
Open Challenges
- Long tail edge cases - autonomous vehicles handle 99.9% of driving well. The 0.1% (construction zones, unusual road markings, unpredictable pedestrians) is extremely hard to solve.
- All-weather operation - heavy snow covers lane markings. Heavy rain degrades LIDAR. Camera-based systems struggle in direct sunlight.
- Regulatory approvals - deploying AV services requires permits from city, state, and federal authorities. Process is slow and varies by jurisdiction.
- Public acceptance - high-profile incidents (Cruise suspension in 2023) damage public trust despite AV safety records that outperform human drivers statistically.