Ego, exo, and the robot data engine

The most important video for robotics isn't of robots. It's of hands.

We wrote about the shape of the modern robot learning pipeline — and about its binding constraint: there is no internet of actions. The field's answer is to manufacture one, and the richest ore body available is video of people doing things with their hands. Billions of hours of it exist; almost none of it was recorded with robots in mind. Turning that video into motor control is now its own discipline, and it splits along a line the field borrowed from the perception community: ego versus exo.

Egocentric video is shot from the actor's own point of view — smart glasses, a head-mounted GoPro. It's close to what a robot's own camera would see: the hands are in frame, the objects are at working distance, the gaze roughly tracks the task. Exocentric video is shot from outside — a tripod, a security camera, essentially all of YouTube. It's the vastly larger pool, and the harder one to use, because nothing about the viewpoint matches what a robot will experience.

Every pipeline in this space is a strategy for paying down two gaps at once. The view gap: an exo camera sees a scene; a robot sees its own workspace. And the embodiment gap: a five-fingered human hand attached to a human arm is not a two-fingered gripper on a robot wrist, visually or kinematically. Ego capture mostly pays the first gap for you, which is why the field has rushed toward it. Nobody has fully paid the second.

The capture spectrum

The data being collected today sits on a spectrum, trading diversity against fidelity.

At the instrumented end sit paired ego-exo rigs — Ego-Exo4D, built by Meta and fifteen university partners, captures the same skilled activity simultaneously from worn cameras and surrounding ones, about 1,300 hours of it with synchronized geometry. Data like this is expensive and comparatively scarce, but it's the training ground for the bridge models — the ones learning to translate between third-person observation and first-person execution.

Then come the glasses. Projects like EgoDex (hundreds of hours of dexterous manipulation shot through Apple Vision Pro) and the multi-institution EgoVerse corpus record ordinary people doing ordinary tasks through head-worn sensors that also log hand pose. This is the workhorse tier: cheap enough to scale, rich enough to learn from.

The clever middle is the instrumented tool. The Universal Manipulation Interface and its descendants put a trigger-activated robot gripper in a human hand, with a wrist camera and SLAM recovering the trajectory. A person sweeps through a kitchen at human speed — but through the robot's own end-effector, so the embodiment gap at the hand simply vanishes. Successors add tactile sensing and bimanual rigs. It's the fidelity of teleoperation at a fraction of the cost, bounded only by how many grippers you can hand out.

And at the far end sits the open internet — the largest pool and, for now, the least mined, because it's exo, unlabeled, and shot by people who had no idea they were generating training data.

Three pipeline families

What happens to this video next falls into three families, and the leading labs increasingly run all of them.

Retarget. The explicit route: recover the hand, map it to the robot. Modern pipelines extract 3D hand pose frame by frame, then convert it to a robot action — a virtual fingertip computed from finger positions becomes gripper width, a grasp frame built from the thumb-fingertip axis becomes end-effector orientation, and the whole trajectory is smoothed before it's usable. The retargeted data then co-trains alongside real robot demonstrations. A careful 2026 study of exactly this recipe found that the unglamorous details carry the result: aligning image scale across cameras, normalizing action distributions per dataset, giving each embodiment its own encoder and decoder rather than sharing weights — and above all, hand-pose quality. Triangulated multi-view hand labels roughly doubled task success over monocular estimates, and adding pose noise degraded transfer monotonically. Their 28 hours of high-quality labels outperformed a dataset five times larger. In this family, quality beats quantity, decisively.

Repaint. The synthesis route: don't map the human to the robot — replace the human with the robot. The most complete version to date, Ego2Robot, runs a full graphics-and- inference pipeline over egocentric video: segment out the human arm, inpaint the background behind it, search for a robot base position from which the motion is kinematically feasible, then render the robot into the frame, depth-ordered, executing the retargeted trajectory. Filter the output three ways — kinematic failures, statistical outliers, and a vision-language model checking that the result still looks like the task. From roughly two thousand hours of human video it manufactured over eighteen thousand hours of synthetic robot data across fifteen different robot bodies. The measured gains are telling: modest on clean benchmarks, largest under lighting, background, and appearance shifts. Repainted data buys robustness — it teaches the policy what doesn't matter.

Latent actions. The implicit route: stop trying to recover actions at all. Latent action pretraining learns a compressed vocabulary of "what changed between these two frames" directly from video — no labels, no hand model — pretrains a vision-language-action model to predict those latent actions, and only at the end grounds the latent vocabulary in real motor commands using a small robot dataset. This is the same idea inside DeepMind's Genie world models, and it's the only family that works on raw exocentric internet video today, because it never needs to see a hand it can retarget. Its known weakness is also instructive: with distracting motion in frame — other people, camera shake, irrelevant objects — the latent vocabulary drifts toward modeling the wrong thing without some supervision to anchor it.

What the results actually say

Strip the papers to their load-bearing findings and a consistent picture emerges, as of mid-2026.

Human video works, and it scales. The first scaling laws connecting egocentric data volume to robot policy quality landed this year — log-linear improvement over tens of thousands of hours — and co-training studies consistently find that at the margin, an hour of human data now buys more generalization than an hour of teleoperation, at a fraction of the cost.

But it works selectively. The gains concentrate where data is scarce and variation is wide: new scenes, new lighting, low-data regimes — one study measured its largest improvement, nearly thirty points absolute, exactly when robot data was most limited. Human video is not yet how policies acquire their hardest skills; it's how they stop being brittle about everything surrounding the skill.

And the engineering is the moat. The same data helps or hurts depending on normalization details, label quality, and sampling ratios that never make the headline. This looks less like a modeling problem than a refinery problem — crude in one end, usable actions out the other, with yield determined by process discipline. The labs winning this race are not the ones with the most video; they're the ones with the best refinery.

What's still open

The exo ocean remains mostly unfished — the view gap is real, and the paired datasets that could close it are young. Video carries no force: the difference between sliding a jar and gripping it white-knuckled is invisible to every camera, which is why tactile-instrumented capture tools are appearing. Curation at internet scale is unsolved — VLM filters are a start, not an answer. And the recipes remain fragile enough that results routinely fail to replicate across labs with the same data.

None of this dims the direction. The field spent years arguing whether robots should learn from human video; the argument is over, and the work has moved to the refinery floor. That's the part we find most interesting — as usual, the leading edge looks less like a breakthrough and more like an engine being assembled, piece by piece, out of problems nobody tweets about.