Ai2 ToyBlocks Release: Robot Learning Testbed, 85,336 Demos

Open Model · ax-trend issue #3847 · Reported November 6, 2026

ToyBlocks title card

Ai2 released ToyBlocks on November 6, 2026, the first environment of its new FoundationWorlds framework. ToyBlocks is an open simulation testbed built around a tabletop robot arm and blocks, covering tasks such as pushing, flipping and stacking. The release ships 85,336 automatically generated long task demonstrations, two trained models, and the full training and evaluation code under the Apache 2.0 license.

The point of the release is control, not scale for its own sake. Robot demonstrations are usually tied to narrow tasks, and a single success rate says little about what a model learned or where it failed. ToyBlocks keeps the world small enough that conditions can be changed one at a time, so a failure can be traced to the plan, to a single skill, or to the transfer from simulation to hardware.

Long task demonstrations in the first release85,336
Total frames across the demonstration set144 million
Robot execution represented in the dataAbout 1,600 hours
Peak simulation speed on a single GPUUp to 24x real time

Release Contents and License

The package covers the whole loop rather than a dataset alone. It includes the simulation assets, the task and demonstration generators, the generated demonstrations, two trained models, training code, evaluation code and a validation set. The two models take different routes to the same tasks: one builds on Molmo 2 in a hierarchical setup, and the other builds on ManiFlow. The code lives in the allenai/ToyBlocks repository on GitHub, created on November 4, 2026, under Apache 2.0.

That completeness matters for evaluation work. A testbed where the data generator, the model and the scorer are all public lets outside teams rerun the same conditions and change one layer at a time. It also means the numbers below can be checked against the released validation set instead of resting on a blog post alone.

Real robot arm and letter blocks on a table in the ToyBlocks transfer test setup
The real robot setup behind the transfer tests: a tabletop arm works on letter blocks while a tripod camera records the scene. (Photo: Hugging Face, Ai2)

Demonstration Generation Without Teleoperation

ToyBlocks demonstrations are made without a human remotely operating the robot. Reinforcement learning and motion planning first produce short term skills, the moves a single push or flip requires. A task and motion planner then sequences those skills into longer tasks, so each long demonstration is composed of about 6 to 12 skill demonstrations.

Diagram of ToyBlocks demonstration generation from short skills to long demonstrations
Demonstration generation in ToyBlocks, from short skills built with reinforcement learning and motion planning to the released long task set.

Hundreds of plans run in parallel on a single GPU, at up to 24 times real time speed. The economics are the argument: teleoperated data costs human hours per trajectory, while planner generated data costs compute, and compute can be rerun with a different task mix. The trade is realism. A planner produces clean, well formed trajectories, and a model trained only on those may meet messier human style variation for the first time on real hardware.

Sim to Real Transfer Results

At the atomic skill level, transfer held up. The ManiFlow model scored 80.0% in simulation and 81.1% on the real robot, so the short skills themselves crossed from simulation to hardware with no measured drop. The longer tasks tell a different story. The hierarchical combination of Molmo 2 and ManiFlow scored 55.5% in simulation, 111 of 200 trials, and 50.0% on the real robot, 15 of 30 trials.

Bar chart of ToyBlocks success rates in simulation and on a real robot
Success rates reported by Ai2 for atomic ManiFlow skills and for the hierarchical Molmo 2 and ManiFlow system, in simulation and on a real robot, on a 0 to 100 scale.

Read together, the two rows locate the weakness. Single skills transfer, and composed long tasks lose ground, which points at planning and error recovery across steps rather than at basic manipulation. That is exactly the split a layered testbed is built to expose, and it is invisible in any benchmark that reports one end to end number.

The real robot figure for the hierarchical system rests on 30 episodes. Ai2 states directly that more trials are needed before that 50.0% rate carries statistical confidence, so the sim to real gap should be read as a signal, not a measurement.

Agent Experiments With Skill Descriptions

Ai2 also ran an agent system based on Claude Opus 5.5 against the environment. Given only text descriptions, the agent solved all 30 classic block stacking tasks. It also solved all 30 in a closed loop condition where it revised its actions while watching camera images. On a harder condition that combined finding blocks with spelling stacks, the agent either solved the task or correctly judged it impossible in 63% of cases.

The description experiment is the more transferable result. When the agent could run its own skill experiments and rewrite the skill descriptions from what it observed, its success rate rose from 24% with the default descriptions to 63%. Descriptions written by human experts reached 36%. In this setup, documentation quality is a performance variable, and descriptions earned by testing beat descriptions written from intuition, including expert intuition.

Open Limits and Missing Pieces

Ai2 is explicit about what ToyBlocks has not solved. Spatial generalization in vision language action models stayed weak in the early results, and Ai2 writes that generalization remains far from a solved problem. A fully integrated reinforcement learning training feature is also absent from the current package, so teams that want to train skills end to end inside the framework have to wait or wire their own loop.

A second release with larger data and more skills is targeted for the coming weeks. It has not been published yet, and the current judgment has to rest on the first release: strong atomic transfer, weaker composed task performance, and an evaluation structure that makes that distinction measurable.

Position in Ai2 Open Robot Work

ToyBlocks follows earlier open robot learning work from Ai2, including MolmoAct 2, MolmoBot and MolmoSpaces, and it sits next to the same institute's evaluation releases in other domains. Its contribution is less the block world itself than the accounting it enables. Task specifications are explicit enough that a failure can be assigned to the layer that caused it, which is the same discipline agent evaluation is still missing in most benchmarks.

For teams building robot learning pipelines, the practical use is immediate: the generators and validation set are public today, so ToyBlocks can serve as a small, fast regression world for planner and policy changes before those changes are spent on real hardware time.

Related Reading

For another recent Ai2 evaluation release, see the earlier report on StateBench, where frontier models fall to 13% on long document state tracking. StateBench stresses state tracking in documents, while ToyBlocks stresses state tracking in a physical task, and both are built to show which layer fails instead of reporting a single score.

FAQ

What is ToyBlocks?

ToyBlocks is an open simulation framework from Ai2, released on November 6, 2026 as the first environment of the FoundationWorlds framework. It pairs a tabletop robot arm with blocks and covers tasks such as pushing, flipping and stacking. The release includes simulation assets, generators, automatically generated demonstrations, two trained models, training code and evaluation code under the Apache 2.0 license.

How are ToyBlocks demonstrations created?

The demonstrations are generated automatically, without human teleoperation. Reinforcement learning and motion planning produce short term skills, and a task and motion planner sequences those skills into long tasks. Hundreds of plans run in parallel on a single GPU at up to 24 times real time speed, and each long demonstration contains about 6 to 12 skill demonstrations.

How well do the ToyBlocks models transfer to a real robot?

Ai2 reports that the ManiFlow model reached 80.0 percent on atomic skills in simulation and 81.1 percent on a real robot. The hierarchical Molmo 2 and ManiFlow combination reached 55.5 percent in simulation, 111 of 200, and 50.0 percent on the real robot, 15 of 30. Ai2 notes that 30 episodes are not enough to establish statistical confidence.

What is missing from the first ToyBlocks release?

A fully integrated reinforcement learning training feature is not in the current package. Ai2 also reports that spatial generalization in vision language action models remained weak in early results. A second release with larger data and more skills is targeted for the coming weeks but has not been published yet.

Sources

Open ModelAgentic BenchmarkAX Trend