Two new judges on robot-eval-bench: Gemini 3.7 flash and Muse Spark 1.1
We added Google's Gemini 3.7 flash and Meta's Muse Spark 1.1 to the benchmark. One of them quietly fixed the biggest weakness in the original results.
Raul Romero
Co-founder, Kite ML
Aug 13, 2026 · 3 min read
When we launched robot-eval-bench, we benchmarked five frontier models as robot-policy judges: can a vision-language model actually tell whether a robot did the task, from keyframes and from video, across 16 open datasets. The headline was that there's no single best judge. The winner flips depending on the approach: Gemini was the keyframes specialist and collapsed on video, while the reasoning models were steadier and got the most out of the full clip.
The two approaches are simple. Keyframes shows the judge four evenly spaced stills from the run; video shows it the whole clip. Same episode, same question, but a judge can be far more reliable on one than the other.
Two ways to show the judge an episode
Since then, two new frontier models shipped, so we ran them through the exact same benchmark: Google's Gemini 3.7 flash and Meta's Muse Spark 1.1, Meta's first paid frontier model. Same 16 datasets, same keyframes-versus-video setup, same ground truth. Here's the updated board. The full data and code are open source, and there's a live leaderboard page.
Bars scaled from 70% to 100% so the spread reads clearly. Hover any bar for the full keyframes / video / cost breakdown. The leader for the selected metric is tinted in its maker's brand color.
Accuracy vs cost per episode
Up and to the left is better: more accurate, cheaper to run. Cost is on a log scale.
Gemini 3.7 flash fixed the video collapse
This is the most important change. Gemini 3.6 flash was the best judge in the field at telling whether a policy succeeded from keyframes (94%). But the moment it had to watch the full video of the episode, its accuracy dropped from 94% to 63%, barely above a coin flip. Whether you could trust its verdict came down entirely to how you showed it the run.
Gemini 3.7 flash closes that gap. It still reads keyframes at 93%, and now judges the full video at 89% too, a 26-point jump over the previous generation. That turns it from a keyframes-only specialist into a judge you can point at either view and trust, and its average ties GPT-5.6 Sol for the best in the table (91%). And it does it at flash-tier cost: grading all 16 datasets came to about $1.80. If you're evaluating policies at any scale, near-top accuracy on both views for a fraction of a cent per episode is hard to argue with.
On video it jumps from 63% with Gemini 3.6 flash to 89%, so the collapse is gone. Its average ties GPT-5.6 Sol for the best in the field.
Muse Spark 1.1 is a video specialist
Meta's Muse Spark 1.1 is a multimodal reasoning model, and it behaves like the other reasoning judges we tested: it reads a policy's success better from video than from keyframes. It lands at 87% on video, in the mix with the best video judges (GPT-5.6 Sol at 92%, Gemini 3.7 flash at 89%), and 83% on keyframes. Same pattern as GPT, Kimi, and Opus, which all prefer the full clip. Being a reasoning model, it spends real tokens thinking before it answers, which makes it the most expensive judge per episode of the group and the slowest to run, so it's a strong-video pick rather than a value pick.
Video edges out keyframes (87% vs 83%), the reasoning-model pattern, and enough to sit alongside the best video judges.
The takeaway holds, and sharpens
The original finding was that the right approach is a property of the judge you picked. That's still true, but the map has shifted:
- Reasoning models still own video, and Muse Spark 1.1 joins that club.
- The value frontier moved. Gemini 3.7 flash gives you ~90% on both approaches for a fraction of a cent per episode, which is a better deal than anything in the first round.
Every per-episode judgment, the ground truth, the cost model, and the code to reproduce all of this are open. Benchmark your own judge against the same episodes:
Explore robot-eval-bench on GitHub
See the full standings any time on the live leaderboard.
