Teleoperation, UMI, and World Models: How Robots Learn From Us

If you're following physical AI from the outside, the vocabulary moves fast: teleoperation, UMI, egocentric data, VLA, world models. They sound like competing buzzwords. They're not—they're pieces of one picture. Here's the field guide I wish I'd had.

Teleoperation: puppeteering a real robot

The workhorse of robot learning. A human operator drives a robot through a task—often with a "leader" arm the human moves and a "follower" arm that mirrors it—while the system records everything: the cameras, the joint positions, the gripper. The open-source ALOHA rig made this the standard; most bimanual manipulation data today traces back to that design.

Teleoperation gives you the gold-standard signal: the exact robot, doing the exact task, with perfect action labels. The catch is cost—you need the robot, the rig, and a human's time for every hour of data.

UMI: a robot gripper in a human hand

UMI (Universal Manipulation Interface) is a clever shortcut. Instead of a robot, you hand a person a handheld gripper with a camera on it. They do the task; visual SLAM recovers the gripper's trajectory and open/close. You get real, robot-shaped action data—without a robot in the loop.

It's cheaper and portable (you can collect in any kitchen or warehouse), and the wrist-camera view matches where a robot's own camera sits. The tradeoffs: it's gripper-only (not a dexterous hand), and there's still a gap between a handheld device and the deployed robot.

Human and egocentric data

The cheapest, most abundant rung: humans doing tasks, captured on a head-mounted camera, sometimes with hand tracking. It scales enormously and covers endless diversity. But it's a bare human hand, not a robot, and there's no clean action signal—you have to estimate what the hands did. Great for teaching general priors; not enough, on its own, to make a specific robot reliable.

Put these three together and you get a ladder: human video (cheap, big gap) → UMI (cheaper than a robot, small gap) → teleoperation (expensive, no gap). More scale as you go down; more fidelity as you go up.

VLA vs. world models: act vs. imagine

Two kinds of model get built from this data, and people often think you must pick one:

• A VLA is a policy—the reflexes. It maps what the robot sees to an action, many times a second. It acts.
• A world model is a simulator—the imagination. Given a state and an action, it predicts what happens next. It predicts.

They're complementary, not rivals. The world model can serve as a training gym (let the policy practice in imagined rollouts), a planner (propose an action, imagine the outcome, pick the best), or a safety check. But only one thing ever moves the robot: the policy. The world model whispers; the reflexes act.

So which do you need?

It depends on the stage. Pretraining wants breadth and scale—cheap video and human data. Post-training wants fidelity—teleoperation on the target robot. World models want mostly cheap video, plus a smaller amount of real robot data to ground their predictions against actual physics.

That last point is the one people miss. Even the "we'll simulate everything" camp needs real demonstrations to keep their world models honest. Simulation scales cheaply, but it's calibrated against reality—and reality only comes from a real body doing a real task.

Which is the through-line of all of it: however clever the models get, they are still learning from us. The demonstrations are the ground truth everything else is built on.