Can a model tell whether a robot did the task? Benchmarking task success evaluation

The hardest question in robot learning isn't "can the policy move." It's "did it actually do the thing." A policy can pick up a cup with picture-perfect motion and still put it down in the wrong spot. If your evaluation can't tell those two runs apart, every number that comes after it is fiction.

And right now, that evaluation is mostly a person with a spreadsheet. Most companies deploying robots run the policy on real hardware and write down, episode by episode, whether the task worked. That's broken. No engineer, and really no one on a team, should have to babysit a robot just to find out whether a task succeeds.

The fix is to let a model do the grading. We've been building an auto-eval harness inside Kite that hands the job to a vision-language model: show it an episode, tell it the task, and ask the only question that matters. Was the task accomplished, yes or no? This post is about how well that actually works, which models are good at it, and a couple of results we didn't expect.

What the judge sees

Every episode is a short clip of a real robot attempting a real task, paired with the exact instruction the judge is asked to grade against. Those instructions come straight from the datasets. A few examples:

• "Pick up the coffee pod, place it in the coffee maker, close the coffee maker, and place a cup in the coffee maker."
• "Cut a piece of tape from the tape dispenser and place the cut piece of tape onto the cardboard box."
• "Tear off a piece of paper towel and place it flat on the table next to the tipped-over can."

Getting those instructions right matters, because some of the labels the datasets ship are just wrong. Take a popular open source data set. The lerobot/aloha_static_towel dataset says the task is cleaning up a spill, but there is no spill in the video. The robot is simply placing a paper towel on the table. Grade the judge against the wrong task and it fails every episode, so we fixed the label to match what the video shows before running the benchmark.

Here's one coffee episode two ways: the full demonstration, which accomplishes the task, and the same episode truncated partway through, which does not.

That pairing points at a real gap in how policies get evaluated. The open-source episodes uploaded to HuggingFace are almost entirely success examples: they're demonstrations, recorded to train from, not a balanced mix of what worked and what didn't. But a judge is only useful if it can tell a failure from a success, so we have to manufacture the failures ourselves, truncating, shuffling, reversing, or swapping the task on genuine demos, to get any negatives to grade against at all.

The judge also never sees the episode as raw footage. It arrives in one of two forms:

• Keyframes. Four evenly-spaced stills pulled from the episode, first and last frame included, leaving the judge to infer the motion in between.
• Video. The full clip: a single natively-encoded MP4 for models that accept video, or a dense stack of about sixteen frames for those that don't.

We then run the same battery of episodes (genuine successes plus the deliberately broken negatives) through both approaches, across five frontier models, and score each on accuracy against a human-reviewed ground truth of 16 datasets.

How each approach reaches the judge

Both approaches start from the same episode and end at the same yes-or-no verdict. What differs is the middle: how the footage is turned into something a vision-language model can read.

Keyframes hand the judge a few high-resolution stills and let it infer the motion in between. Video hands it the whole trajectory: a single natively-encoded clip for models that accept video, or a dense stack of frames for those that don't. That one branch in the pipeline is what makes both accuracy and cost swing so hard from model to model.

The benchmark

Five judges: Gemini 3.1 Pro, Gemini 3.6 Flash, Claude Opus 5, GPT-5.6 Sol, and Kimi 3. Accuracy is the fraction of episodes the judge labeled correctly; the best judge in each column is tinted green.

A few things jump out.

There is no single best judge; there's a best judge per approach. Gemini 3.6 Flash is the strongest keyframe reader in the field at 0.94, and then falls to 0.63 on video. GPT-5.6 Sol is the opposite kind of judge: good on keyframes (0.90) and the best of anyone on video (0.92). Pick the wrong one for your approach and you've hobbled your evaluation before it starts.

Bigger isn't better. Gemini 3.6 Flash beats Gemini 3.1 Pro on both keyframes (0.94 vs 0.88) and video (0.63 vs 0.59). The smaller, newer model is simply the better judge here; paying up for "Pro" bought worse evaluations.

GPT-5.6 Sol is the all-rounder. It's the only judge that stays strong on both approaches, and averaged across the two it's the strongest judge overall (0.91). If you want one judge you don't have to think about, it's this one.

Kimi 3 is the value pick. It lands at 0.86 and 0.84, a hair behind the leaders, while costing a fraction of what they do to run. More on that below.

The first surprise: accuracy flips with the approach

The intuitive story is "more frames, better judgment." Video should dominate keyframes everywhere. It doesn't. Instead the ranking inverts by model family:

• Reasoning models do well on video. GPT-5.6 Sol, Kimi 3, and Claude Opus 5 all stay within a few points across approaches, and GPT actually posts its best score on video. They use the extra frames to reason about the motion.
• Gemini collapses on video. Both Gemini judges are excellent on keyframes and then crater on the full clip. Flash drops from 0.94 to 0.63, Pro from 0.88 to 0.59. Handed dense video, they lose the thread instead of gaining signal.

So there's no single right approach. It depends on the judge. Give Gemini a few keyframes and it does great; give a reasoning model the full video and it does great. Match them the wrong way around and you can drop a judge from 0.9 accuracy to 0.6, without changing anything about the robot itself.

The second surprise: cost flips too

Cost runs the same way, and the direction depends entirely on whether the model ingests video natively.

For Gemini, video is the cheap option: one native clip costs almost nothing (about $0.0005 an episode), while a few full-resolution keyframes cost about ten times more. For the other models it flips, because they read video as roughly sixteen separate frames. Claude Opus 5 goes from $0.0107 on keyframes to $0.0648 on video, and GPT from $0.0078 to $0.0538. Kimi 3 is the cheapest reasoning judge either way.

Keyframes are where most evaluation actually runs, so it's worth looking at that approach on its own. Plotting accuracy against cost per episode, two judges sit on the efficient frontier: Kimi 3, the cheapest, and Gemini 3.6 Flash, the most accurate. Everything else is dominated, Claude Opus 5 most of all, paying the most to score the worst.

Put accuracy and cost together and the call is easy. For most cases I'd reach for Kimi 3: almost the best accuracy, and by far the cheapest reasoning judge. Want the highest accuracy and don't mind paying for video? That's GPT-5.6 Sol. And Gemini 3.6 Flash on keyframes is a great cheap option, as long as you never hand it the full clip.

Why this matters beyond a leaderboard

Everything above is in service of one goal: knowing whether a robot will work before it is ever deployed.

Real-world evaluation is slow, expensive, and dangerous to get wrong. You can't run a policy a thousand times across a thousand lighting conditions and kitchen layouts on physical hardware, but you can in simulation, if you have a judge you trust to score every one of those runs honestly. A benchmarked VLM judge turns "we think it works" into a measured number, across far more conditions than a human could ever sit through.

That's the loop we're building toward: generate varied simulated worlds, run the policy across all of them, and let a benchmarked judge tell you exactly where it succeeds and where it breaks, before any of it touches real hardware.

We're looking for design partners

We're opening up evaluation-in-simulation to a small group of design partners. If you're training robot policies and want an honest verdict on whether they work before they're ever deployed, we want to build it with you.

Sign up as a design partner→