The evidence for data augmentation in robotics

Every robotics team hits the same wall. The policy works in the room where the data was collected. Move it to a different kitchen, change the lighting, put a different mug on the table, and it falls apart.

The obvious fix is more data. The problem is that robot data costs robot time — a human, a rig, and hours per hundred episodes. So the field spent the last nine years asking a cheaper question: how much of that generalization can you buy by changing the data you already have?

The answer, across a lot of independent groups, is: a surprising amount. Here is what the results actually say.

It started with textures that looked wrong

The founding result is Tobin et al. (2017). They trained an object detector entirely in simulation, on scenes with deliberately non-realistic random textures — no real images at all — and it localized real objects to about 1.5 cm on average, well enough to drive real grasping. The insight was counterintuitive: you don't make the simulator look real, you make it look varied enough that reality is just another variation.

Two follow-ups sharpened it into a number worth quoting. In RCAN (James et al., CVPR 2019), a translator trained with zero real data hit 70% zero-shot grasp success on unseen objects — which the authors describe as almost double what the same task achieved on domain randomization alone. Fine-tuned with 5,000 real grasps, it reached 91%, comparable to a system trained on 580,000 real grasps.

RL-CycleGAN (Rao et al., CVPR 2020) is the cleanest data-efficiency comparison in the literature. Real data only, 5,000 episodes: 15% success. Same 5,000 episodes plus a task-aware sim-to-real translation: 75%.

In visual RL, augmentation was the algorithm

Around 2020 two papers landed within days of each other and made the same embarrassing point.

RAD (Laskin et al., NeurIPS 2020) and DrQ (Kostrikov et al., ICLR 2021) showed that plain image augmentation — random crops, random shifts — bolted onto a standard RL algorithm beat the elaborate machinery the field had built to avoid needing it. No world model, no contrastive objective, no auxiliary loss. RAD reported state-of-the-art on all six DeepMind Control tasks at both benchmark budgets. DrQ beat CURL, the leading contrastive method, on all six tasks at both 100k and 500k steps.

RAD also ran the comparison that should worry anyone budgeting a data collection: on ProcGen's BigFish, its random-crop agent trained on 100 levels outscored the unaugmented baseline trained on 200. Augmenting was worth more than doubling the training distribution.

This isn't just an RL curiosity. Robomimic (Mandlekar et al., CoRL 2021) found the same thing for imitation learning on real hardware: on a real Franka pick-and-place task, dropping pixel-shift randomization took a proficient 73.3% policy down to 26.7%. Same data, same architecture, same demonstrations — one augmentation removed.

Generative augmentation: new scenes without new robot time

Once image models got good, the augmentation stopped being crops and started being worlds.

ROSIE (Yu et al., RSS 2023) inpainted new objects, backgrounds, and distractors into existing RT-1 episodes with text guidance and no new robot data. Picking objects in new backgrounds went from 0.33 to 0.71; picking novel objects from 0.25 to 0.75; a task the baseline never once completed reached 0.60.

GenAug (Chen et al., RSS 2023) is the one I find most striking about cost. Ten demonstrations per task, all collected in a single environment, augmented into a thousand generated variations — 85% average success across unseen real environments.

The trend continues into the VLA era. DreamGen (Jang et al., CoRL 2025) used a video world model to synthesize training data from a single pick-and-place teleoperation dataset, and took GR00T N1 from 11.2% to 43.2% on fourteen behaviors it had no real demonstrations of. In environments it had never seen, the baseline scored zero on all thirteen tasks; with generated data, 28.5%.

Diversity is the thing you're actually buying

The most useful framing came from ablations that separate how much data from how varied it is.

RT-1 (Brohan et al., RSS 2023) ran it directly: cutting 25% of the tasks while keeping 97% of the episodes cost about as much generalization as throwing away 49% of the data outright. Their conclusion — "data diversity is more essential than data quantity."

Data Scaling Laws in Imitation Learning (Lin et al., ICLR 2025) put a curve on it. Generalization follows a roughly power-law relationship with the number of environments and objects. Demonstrations per environment saturate: past a threshold, more episodes in the same room buy almost nothing.

That reframes what augmentation is for. It isn't a way to fake having more data. It's a way to move along the axis that actually pays — scene and object diversity — without buying more robot time.

There's a failure-mode version of the same story. Xing et al. (CoRL 2025) built datasets where camera viewpoint was deliberately correlated with the target object, and showed that π₀ learned the shortcut: it followed the viewpoint and ignored the language instruction. Targeted augmentation measurably reduced that shortcut reliance. Augmentation isn't only additive — it breaks correlations you didn't know your dataset had.

Where augmentation doesn't help

Three honest limits, all from the same literature.

It stops mattering once you have enough real data. RL-CycleGAN's enormous advantage at 5,000 episodes shrinks to about 7 points at 580,000. Augmentation buys the most exactly when real data is scarce — which is most teams, but not all.

Fancier is not better. GreenAug (Teoh et al., 2024) ran a large real-robot comparison — 8 tasks, 3 novel scenes, ~112 evaluation episodes per task-method pair — and found chroma-keying random textures scored 91%, beating both diffusion-generated photorealistic backgrounds (75%) and their own generative variant (77%). The authors' read is that specific semantic content isn't what makes it work.

A bad augmentation does nothing at all. In that same table, one GreenAug variant scored 58% against 55% for no augmentation whatsoever. Which augmentation you apply matters more than whether you augment.

And it supplements real data rather than replacing it. DreamGen's own RoboCasa numbers make the point cleanly: 300 real demonstrations scored 49.6%, real plus 240k generated trajectories scored 57.6% — and generated trajectories alone scored 20.6%. Augmentation multiplies a real dataset. It doesn't conjure one.

What this means for how we built Augment

We read this literature as a fairly specific instruction: generate scene and object diversity, keep the trajectory intact, and make it cheap enough to do routinely rather than once.

That's what Kite's Augment does. You point it at a LeRobot dataset you already have, describe the variation in plain language, and it regenerates the pixels while carrying the original joint and action trajectory onto the new footage. The output is a standard LeRobot dataset that drops into the training pipeline you already run.

The one design choice worth calling out is per-episode prompts. Ten episodes generated from one instruction is ten samples of a single world; ten different instructions is ten worlds. Given what RT-1 and the scaling-laws work say about diversity being the axis that pays, that distinction is most of the value — so variation_prompts takes one prompt per episode.

What we won't do is quote you a success-rate number. None of the results above transfer directly to your robot, your task, or your camera placement — the honest version of this literature is that augmentation reliably helps when real data is scarce and scene variation is what's missing, and that the size of the effect is something you have to measure on your own setup. Which is the real argument for making augmentation a call you can put in a loop: augment, train, evaluate, look at what still fails, and augment for that.

Augment a dataset→

Augment runs in the app and on the Platform API — the API reference and quickstart are at docs.kiteml.com.