Robot Software Stack

A robot is a system of systems. Unlike a web application where you have a single request–response loop, a robot runs multiple concurrent software layers, each with different timing requirements, abstraction levels, and failure modes. Understanding this stack is essential before integrating AI into physical systems.

The Five Layers

┌──────────────────────────────────────────┐
│  Task Layer                               │
│  "Pick the red box and place it on        │
│   the shelf in bay 3"                     │
├──────────────────────────────────────────┤
│  Planning Layer                           │
│  Decomposes task into sub-goals:          │
│  1. Locate red box (vision query)         │
│  2. Navigate to box (path planning)       │
│  3. Grasp box (manipulation)              │
│  4. Navigate to shelf                     │
│  5. Place with correct orientation        │
├──────────────────────────────────────────┤
│  Perception Layer                         │
│  - Camera → object detection + 6D pose   │
│  - Depth sensor → 3D scene map           │
│  - Force/torque → grasp quality          │
│  - IMU → robot orientation               │
├──────────────────────────────────────────┤
│  Control Layer                            │
│  - Inverse kinematics (joint angles)     │
│  - Trajectory planning (collision-free)  │
│  - PID / impedance controllers           │
│  - Safety monitors (force limits)        │
├──────────────────────────────────────────┤
│  Hardware Layer                           │
│  - Actuators (motors, servos, tendons)   │
│  - Sensors (cameras, lidar, IMU, F/T)    │
│  - Onboard compute (edge GPU / NPU)      │
└──────────────────────────────────────────┘

Task Layer - Natural Language Interface

The task layer is where LLMs enter the robot stack. A human (or an orchestration system) specifies what should happen in natural language. The LLM at this layer performs:

• Task decomposition - breaking "clean the table" into ordered sub-tasks
• Common-sense reasoning - knowing that you pick up objects before moving them
• Error recovery - if the gripper misses, re-plan rather than stop
• Ambiguity resolution - asking for clarification when the instruction is underspecified

This layer runs at human timescales - seconds to minutes. It can afford to call a cloud API.

Planning Layer - Task to Motion

The planning layer translates high-level sub-goals into concrete motion sequences. Two key planning problems:

• Task and Motion Planning (TAMP) - combined symbolic reasoning (what to do) and geometric reasoning (how to move). Classic robotics problem; AI is now replacing hand-crafted planners here.
• Path planning - finding a collision-free trajectory from current state to goal state. Algorithms: RRT, RRT*, A* in configuration space. Neural planners are emerging.

Runs at 1–10 Hz. May run locally on the robot for latency-sensitive tasks.

Perception Layer - Seeing the World

The perception layer converts raw sensor data into a structured representation the planning layer can reason over. Key perception tasks:

• Object detection and 6D pose estimation - "Where is the red box, and which way is it oriented?" 6D pose (3D position + 3D orientation) is needed for grasping.
• Scene segmentation - identifying traversable floor vs. obstacles vs. objects of interest.
• Depth estimation - converting 2D camera images to 3D point clouds, or using dedicated depth cameras (Intel RealSense, structured light).
• Grasp quality estimation - using force/torque sensors to detect slip or measure grasp firmness in real time.

Runs at 10–30 Hz. Must be fast enough to react to a moving environment.

Control Layer - Joints and Motors

The control layer converts desired end-effector positions into actual motor commands. This is classical robotics, but it must be robust to the AI layers above it:

• Inverse kinematics (IK) - given "put the gripper at position X with orientation Y," compute what joint angles achieve this. Non-trivial for redundant robots (7+ DoF arms).
• Impedance control - rather than commanding exact positions, control the robot's effective stiffness. Makes contact tasks safer - the robot "gives" when it hits something unexpected.
• Safety monitors - hard limits on joint torques, velocities, and proximity to humans. These run at 1 kHz and cannot be overridden by higher layers.

Runs at 100–1000 Hz. Must be deterministic and cannot afford cloud round trips.

ROS - The Middleware That Glues It Together

The Robot Operating System (ROS) is the de facto middleware for robot software. Despite the name, it is not an OS - it is a publish/subscribe message-passing framework that allows the layers above to communicate without tight coupling.

• Each layer publishes to topics (e.g., /camera/image, /joint_states, /cmd_vel)
• Other layers subscribe and process asynchronously
• ROS 2 adds real-time guarantees, security, and better multi-robot support over the original ROS

Most commercial robots expose a ROS interface, making it the integration point for AI components.

Where AI Plugs In

AI is not replacing the robot stack - it is augmenting specific layers:

• LLMs at the task layer for flexible instruction understanding
• Vision-language models at the perception layer for open-vocabulary object recognition
• Learned policies (neural networks) at the planning layer for dexterous manipulation
• Classical control algorithms remain at the control layer for safety and determinism