LLMs as Robot Brains

Before large language models, programming a robot to complete a new task required substantial engineering: defining states, writing motion primitives, handling edge cases, and testing exhaustively. Each new task was a new engineering project. LLMs have fundamentally changed this by providing robots with something that previously had to be hand-coded: general-purpose reasoning.

The Shift: From Rules to Reasoning

How an LLM Controls a Robot

The LLM does not directly output motor commands. Instead, it acts as an orchestrator that calls robot capabilities as tools - the same tool-calling pattern you use for AI agents:

User: "Make me a cup of coffee"

LLM reasoning:
1. navigate_to(location="coffee_machine")
2. pick_up(object="coffee_pod", type="medium_roast")
3. insert(object="coffee_pod", target="machine_slot")
4. pick_up(object="empty_mug")
5. place(object="mug", target="drip_tray")
6. press_button(button="brew_start")
7. wait_for(condition="brewing_complete", timeout=120)
8. pick_up(object="full_mug")
9. deliver_to(location="user_desk")

Each function call goes to the robot control layer.
The LLM monitors results and re-plans on failure.

SayCan - Grounding LLMs in Robot Capabilities

A landmark result from Google Robotics (2022): SayCan showed that the right approach is to combine LLM knowledge with robot affordances. The key insight: an LLM might suggest "fly to the shelf" - which is physically impossible for a ground robot. SayCan weights LLM suggestions by the robot's estimated ability to execute them:

Score(action) = P(action | language instruction)  ← LLM plausibility
              × P(success | action, robot_state)  ← Robot feasibility

The robot only selects actions it can physically execute.

This grounding prevents the LLM from hallucinating physically impossible plans - a critical safety mechanism.

Code as Intermediate Plans

Rather than calling tools one at a time, some systems have the LLM generate a small program (Python) that orchestrates the robot. This allows loops, conditionals, and error handling - things a linear tool-call sequence cannot express:

# LLM-generated robot program
def sort_objects():
    objects = robot.detect_objects()
    for obj in objects:
        if obj.color == "red":
            robot.pick(obj)
            robot.place(target="red_bin")
        elif obj.color == "blue":
            robot.pick(obj)
            robot.place(target="blue_bin")
        else:
            robot.log(f"Unknown color: {obj.color}")
            robot.place(target="unsorted_bin")

Vision + Language: VLMs as Robot Brains

Pure text LLMs need structured inputs from the perception layer ("I see: red box at position 1.2m, 0.3m, 0.8m"). Vision-Language Models (VLMs) like GPT-4V, Gemini, and Claude can directly process camera images, removing this translation step:

• The robot sends a camera frame directly to the VLM
• The VLM describes what it sees and reasons about next actions
• No hand-coded object detection pipeline needed for task planning

This is the path toward VLA (Vision-Language-Action) models, which go one step further and output actions directly.

Limitations and Failure Modes

• Hallucinated plans - LLMs may suggest actions that are physically impossible without proper grounding (SayCan-style feasibility checks help).
• Latency - calling a cloud LLM adds 200–2000ms latency. Acceptable for high-level planning; catastrophic for real-time control. Hybrid local/cloud architectures are the solution.
• Safety - LLMs have no inherent understanding of physical danger. Explicit safety monitors at the control layer must override any unsafe commands from the LLM layer.
• Context length - long manipulation tasks can exceed the LLM's context window, causing it to "forget" earlier steps. Structured state tracking is required.