Robot 101 · Chapter 06

How robots learn: imitation, data, and the loop

Imitation learning: teaching a robot by demonstration

Behaviour cloning (BC) is the simplest form of imitation learning: record a human performing a task, train a policy to predict the demonstrator's action from the same observation, and run that policy on the robot. No reward function, no physics simulator, none of the instability of reinforcement learning from scratch. The catch is that BC is only as good as its training distribution. If every demonstration starts with an object centred on a counter, and the robot meets that object offset 15 cm to one side, the policy has never seen that state — a small error nudges it into an even less familiar one, and the failure compounds.

DAgger (Dataset Aggregation) is the standard fix: while the trained policy runs, a human expert periodically supplies the corrective action for its current state — not the action the robot was about to take. Those corrections are folded back into training. After a few rounds, the training distribution covers the error states the robot actually meets in the field, not just the clean states a demonstrator chose to show it. In practice, every new deployment site or task variation is a new distribution: a handful of correction rounds — often a couple of hours of an expert's time — can adapt a policy trained elsewhere to a new layout, lighting, or object variation, and each site's corrections make the shared policy better for the next one.

ACT and Diffusion Policy: two ways to plan the next move

ACT (Action Chunking with Transformers) was an influential early policy architecture for manipulating flexible objects consistently — folding cloth, inserting small parts with both hands — from camera images alone, no point clouds required. Its key idea is action chunking: instead of predicting one action per timestep, it predicts a short sequence of future actions (a "chunk") and executes it while the next chunk is computed in parallel, which avoids much of the compounding error plain BC suffers from. ACT also uses a conditional variational autoencoder to handle tasks with more than one valid solution — there are several ways to fold a shirt, and the model commits to one and executes it consistently rather than averaging them into a physically meaningless motion.

Diffusion Policy takes a different route, modelling the action as a denoising process — the same mathematics behind image-generation models. It starts from random noise in action space and iteratively denoises it, conditioned on the current camera image and the robot's own joint state, producing a smooth trajectory. It represents genuinely multi-modal choices (which hand goes first, which side of an object to grasp) better than chunked transformers can, because diffusion doesn't average across valid modes. The cost is speed: it can require many denoising steps and may be too slow for tight control loops (in some reported setups, adding up to roughly 200 milliseconds of inference against a 30 Hz control loop) unless accelerated or distilled — so it's often distilled into a single-step sampler, trading a little representational richness for faster inference.

Teleoperation data collection and the ALOHA lesson

Model architecture matters less than the data it is trained on. Since around 2022, benchmark after benchmark has found that a few thousand high-quality demonstrations beat tens of thousands of mediocre ones. That lesson traces to ALOHA (a low-cost, open-source bimanual teleoperation rig from Stanford) and the subsequent Open X-Embodiment dataset — a Google DeepMind-led collaboration spanning 22 robot bodies and over a million recorded demonstrations. The collection method shapes the data distribution, and the data distribution shapes what the policy can and cannot do.

The ALOHA-style approach is leader-follower teleoperation: two matching small arms, where a human physically moves the "leader" and the "follower" — the one that will deploy — mirrors its joint angles in real time over a low-latency link. Because the operator moves the arm the way they'd move their own, rather than fighting a joystick, trained operators in reported ALOHA-style setups have hit 85-95% completion on well-defined tasks. A complete rig in such setups has run roughly $8,000-$15,000 in off-the-shelf arms and a small recording computer. VR teleoperation (headset plus hand controllers) is the usual alternative for mobile-base navigation and coarse task sequencing — faster to drive, but hand tremor and latency make it a poor fit for sub-centimetre manipulation.

Why many demonstrations get thrown away

A raw teleoperation recording is not training data — it has to pass a quality pipeline first. Across large-scale collection efforts, a recurring pattern has been reported: a meaningful share of raw demonstrations get rejected — commonly cited figures fall in the rough 20-40% range — where the operator lost control, the object slipped mid-task, or the task simply wasn't completed in a reasonable time. Including those failures in training data doesn't just waste storage; it can teach the policy failure modes alongside successes, degrading final performance relative to a clean dataset. Typical rejection checks include: did the task actually complete (verified against a goal image); did it finish within roughly twice the median expert time; did any joint torque spike high enough to suggest the operator was fighting the robot's dynamics; did the gripper leave the intended workspace; and are the camera frames sharp enough to use. Demonstrations that fail on completion or timing are simply discarded; borderline cases go to a human reviewer, who can trim the hesitant lead-in of a recording or discard it outright.

What survives is annotated automatically — object labels, the frames where contact begins and ends, a phase segmentation (approach/grasp/transport/release), and a language instruction — then stored in a standard cross-robot data format that feeds directly into common training frameworks. Unglamorous and labour-intensive, but this stage is what actually determines whether the resulting policy is any good.

Reinforcement learning with safety constraints

Imitation gets a policy to a competent starting point, but reinforcement learning (RL) pushes performance past what any demonstrator achieved. For mobile-base navigation, PPO (Proximal Policy Optimization) is standard — stable to train, and commonly used in legged-robot locomotion research and some deployed controllers. A typical reward is simple: a bonus for reaching the goal, a penalty for a collision, a small time penalty, and a bonus for smooth rather than jerky velocity. For contact-rich manipulation — opening a door, or anything where robot and environment physically push back — SAC (Soft Actor-Critic) is usually preferred, since it is more sample-efficient and explores more broadly than on-policy methods like PPO.

The harder problem is safety. Left unconstrained, an RL policy will happily discover that faster motion earns more average reward, even if an occasional abrupt movement is unwelcome around people. Safe RL adds hard constraints on top of the reward — a cap on end-effector velocity near a person, a cap on joint torque during contact, a minimum stand-off distance on approach. The standard implementation is a constrained Markov decision process with Lagrangian relaxation: a penalty is added to the reward for any constraint violation, and a separate optimiser tunes how heavily that penalty is weighted. This framework, or close variants of it, underlies safety-constrained controllers across the industry.

The deployment loop: shadow mode, versioning, and continuous retraining

A deployed policy is never a finished artefact — it needs to keep improving as new field data accumulates, and the infrastructure to do that safely looks more like software MLOps with extra guardrails than a one-off training run. Every checkpoint is stored in a versioned model registry; a major version bump requires passing a full validation suite across all trained tasks at a high success-rate threshold, a minor version needs task-specific validation, and a patch only needs regression tests on representative demonstrations.

Before any new version reaches the full fleet, it typically runs in shadow mode: deployed to a small subset of robots where it computes actions from live sensor input but never actually executes them — the production policy still drives the robot. A comparison process logs where the shadow policy and the production policy disagree, and a human reviews the more significant disagreements. If the shadow policy's proposed action would have been better, that's a case for faster rollout; if it would have been worse or unsafe, the issue goes back to the model team before the new version ever touches production behaviour. A typical weekly cadence looks like: early in the week, run the previous week's execution logs through the quality pipeline and add newly accepted demonstrations to the training set; mid-week, fine-tune the policy on the updated dataset; later, run automated validation; and only then, if validation passes, push to shadow deployment. Each site that contributes clean data through this loop makes the shared policy a little better for every other site running it.