ToyBlocks makes a robot’s next failure a question you can investigate

The interesting experiment starts after the block falls

A robot tips over a stack. Watching the clip, you might blame a clumsy movement. But the arm may have executed an impossible instruction exactly as requested. The planner may have chosen the wrong block, or a camera view may have hidden the support beneath it. A failure at the end of an action sequence tells you that something went wrong. It does not identify which part should change.

Ai2’s November 6 ToyBlocks release is built around making those questions easier to investigate. Its simulated tabletop combines an arm and marked blocks with tools for generating tasks and demonstrations. The framework separates high-level choices from low-level movements and provides reference planning and action components for comparisons. Its controlled scenes can vary factors such as block geometry and appearance. Ai2 presents it as a setting for studying robot learning and generalization; a result inside this world does not establish that a robot can handle an arbitrary physical workplace.

The useful idea is accessible even if you never train a robot. Instead of asking whether a system can complete one impressive sequence, ask which deliberately changed version of that sequence it can still complete. Then ask what the difference tells you. A small world becomes informative when you can hold some conditions steady, change another, and preserve enough of the run to challenge your explanation.

Source: Ai2: ToyBlocks, a controlled world for robot learning and evaluation ↗ · Ai2: ToyBlocks source repository ↗

Give the imaginary task two different jobs

Consider a fictional sorting station with three wooden cubes. Each cube has a different symbol on one face. The task is to reveal the symbols and arrange the cubes in a specified left-to-right order. This is an illustrative study design, not an experiment conducted for this article, and it does not prescribe a physical robot exercise.

There are two jobs here. One is choosing what to do next: inspect a face, select a cube, or move it to a particular location. The other is carrying out that choice: turn the selected cube and stop with a readable face, or move it without disturbing its neighbor. Separate the records for those jobs. Save the scene observation available before each choice, the selected action and target, and the resulting observation. A single video may contain this information visually, but a structured record makes it easier to line up comparable moments across runs.

Now suppose a cube ends in the wrong place. Ask whether the planner chose that destination or whether the executed movement missed the requested destination. If the selected destination was wrong, more precise control alone will not repair the choice. If the destination was right but the movement failed, a longer verbal explanation of the ordering rule may not help. If the planner never got a readable view of the symbol, neither category captures the whole problem: the observation that informed the choice needs attention.

Make space for more than one explanation. A slightly inaccurate turn might leave a symbol difficult to read, which then prompts a bad choice. Calling the whole episode a reasoning failure would hide the earlier control issue. Calling it solely a control failure would hide the planner’s response to uncertain evidence. A useful diagnostic record shows that chain instead of forcing the run into one permanent label.

Construct pairs that disagree with your favorite explanation

Start with a suspected cause and ask what should happen if it is right. Suppose your working explanation is that the model associates a particular symbol with the cube nearest the camera. Create a paired scene in which the symbols exchange positions while their appearances, requested order and available actions stay the same. If the error follows the position rather than the symbol, the result is consistent with the suspicion. It is not proof: another position-dependent factor could still explain it. The next pair can distinguish those alternatives.

A different pair can keep the task and cube positions unchanged while altering only the camera view. That may help investigate whether an apparent ordering error depends on what was visible. But moving the camera can change several visible properties at once: apparent size, overlap and orientation. Name those consequences before reporting that you changed only one thing. “One camera parameter changed” and “one perceptual difficulty changed” are not equivalent claims.

For a control question, imagine keeping the selected action and target fixed while varying a physical parameter within the simulator. The comparison asks whether the movement tolerates that variation. It does not ask whether the planner can recognize when a different movement is required. Those are separate studies. If you let the planner adapt on every trial, a successful adaptation may conceal a brittle movement; if you freeze the plan, a failure may arise because the fixed plan no longer makes sense.

Keep an unchanged partner for each modified scene. Use several starting layouts rather than judging an explanation from one convenient pair. Record the generator settings and any random seed needed to recreate them. If a modified scene is no longer solvable under the permitted actions, put it in a separate group. Otherwise, your supposed test of robustness may simply punish a model for failing to perform an impossible task.

Borrow a reference component carefully

ToyBlocks’ separation of high-level and low-level components suggests a diagnostic comparison: substitute a reference for one component and observe what remains. In the imaginary sorter, give the learned controller a known valid sequence of action requests. In another condition, let the learned planner choose requests but have a reference executor carry them out. Keep a condition with both learned components as well. These are proposed comparisons, not results reported here.

If the learned controller struggles even with valid requests, that is a reason to investigate execution in those conditions. If the learned planner still gives poor requests when execution is reliable, investigate its choices and observations. If both isolated comparisons look good but the combined system fails, inspect the interface between them. The planner may assume a turn exposes a face that the controller considers optional. Or the controller may signal completion before the image the planner receives reflects the completed movement.

A reference is useful because of a specific promise. Write down that promise. It might provide a valid action sequence for a known scene or execute a narrowly defined skill. It does not necessarily provide a fair replacement for every capability of the learned component. A reference with access to exact simulated state can answer questions that an image-driven system cannot. That extra information is valuable for diagnosis, but the comparison must identify it.

Avoid subtracting two overall scores and naming the difference “the cost of planning.” Components interact. Better execution can produce better observations, while a more conservative planner can make execution easier. A component substitution changes the conditions faced by the rest of the system. Explain which inputs and opportunities changed alongside the component, then describe the result as evidence for a narrower hypothesis. That account is more useful to someone deciding what to improve next.

Vary what training teaches and what evaluation asks

A controlled world also lets you distinguish changing the test from changing the training experience. In the sorting example, one training mixture might mostly show cubes with their symbols already visible. Another might include more episodes in which the system must reveal a hidden face. If you want to learn whether those experiences improve information gathering, preserve the same evaluation scenes and compare the resulting systems there. Otherwise a friendlier test may look like a training benefit.

Keep the amount of training data explicit. Adding extra hidden-face episodes changes both the mixture and the total. Replacing some visible-face episodes keeps the total fixed but asks whether a tradeoff helps. Both are legitimate questions; they answer different things. If compute or training duration also changes, record that rather than implying that the composition alone explains the outcome.

Design evaluation groups before inspecting the final results. One might contain unfamiliar layouts made from familiar symbols. Another might use a new symbol appearance while retaining familiar geometry. A third could combine those changes. The first two help locate a weakness; the third asks how changes interact. A system may tolerate each isolated difference and still fail when they occur together. Conversely, a combined scene may accidentally make a task easier by exposing a previously hidden face.

Be precise about what was withheld. A new arrangement of known cubes is not the same as an unfamiliar cube type. A location between familiar training positions differs from a location beyond their range. Naming the held-out property makes “generalization” a concrete statement. Readers can then decide whether the result addresses the variation they actually care about, rather than assuming that an unfamiliar example stands for every unfamiliar situation.

Turn an experiment plan into an honest explanation

Before running anything, draft the record you would want to read afterward. State the question, the paired conditions, the components that receive the same inputs and the information available to each. Define which observation would support your explanation and which would make you reconsider it. Add a place for runs that cannot be interpreted cleanly, such as a simulation reset or an unavailable action. This is a plan for preserving evidence, not a promise that the result will fit a tidy story.

For the imaginary sorting station, a useful outcome summary could group runs by whether the symbol was revealed, whether the next requested action was appropriate, and whether execution reached its requested result. Include the number of runs behind each group. This allows a reader to see whether an apparently large improvement rests on many episodes or a handful. Retain disagreements and awkward cases. A cube that reaches the right place after the wrong request is especially informative: the final arrangement alone would reward a mistake.

The repository is a starting point for exploring the framework, not evidence that an experiment has been reproduced. If you use it, record the version and relevant settings with your study. Check its current installation and resource requirements before committing to a run. Someone learning the method can instead work through a handful of existing episodes and make a paper diagnosis, clearly labeled as an interpretation of those recordings. That exercise can teach the distinction between choosing and executing without pretending to measure a model.

There is also a boundary to keep around the conclusion. A simulation supports claims about the tested models, scenes and assumptions. Moving to physical equipment introduces questions the screen cannot settle merely by looking realistic. Do not turn a successful simulated study into a claim that a particular desktop arm will work with the framework or reproduce the result. Exact hardware, software and operational requirements need their own verification.

The payoff of a small controlled world is a better next question. Instead of “the robot failed,” you might ask whether it would have succeeded with the same choice and more reliable execution, or whether a changed view would have led it to choose differently. Those counterfactuals give a failure explanatory value. They also make a success more informative: you can say which deliberate variations it survived, and which remain to be investigated.