Claude plays robotics

Do language models’ strengths transfer to robotics, a domain which requires the synthesis of logical skills and precise 3D understanding? Can a model perceive a scene, understand a particular robot’s state, and issue actions that reliably effect change in the physical world?

We ran tests to find out. We gave several language models control over a range of robot bodies—including classic control toys, a simulated quadruped and humanoid, a robotic arm, and a real Unitree Go2 (the quadruped robot of Project Fetch). We gave the models a range of ways to control them, which varied in their abstraction (that is, how “high-level” their instructions are): from directly commanding motor torques (at the least abstract end), to writing controller code, to training a controller from scratch with reinforcement learning, to providing high-level steering instructions to a pretrained robot policy (a separate neural network that turns high-level commands into coordinated joint movements). We tested models’ performance in three areas: on classic control problems (like balancing a pendulum), locomotion and navigation (getting legged robots to balance, walk, and move through space), and manipulation (using a robotic arm to grasp and move objects).

Models are getting better at robotics quickly, but we found that how capable they are depends heavily on how they are connected to the robot—which of the control methods they used. When they must drive the joints themselves they mostly fail. But when they supervise a pretrained controller or use simple orientation tools, they can complete real navigation and manipulation tasks. Some forms of embodiment remain unwieldy and difficult to control, but newer models, especially, are substantially stronger at adjusting their strategies and converting image and sensory understanding into appropriate actions across domains.

This has important implications for the safe development and deployment of language models. Today’s frontier models cannot control humanoid robots without a pretrained policy, but newer models have made real gains in direct manipulation and high-level policy control across the humanoid and quadruped embodiments we tested. We expect future models to be even better. Put concretely: a general-purpose chat model with no robotics training can already, on a good run, write and download its own tools to slowly walk a quadruped through a maze or pick a plate off a counter and set it on a stove, and the gap in reliability is closing with each model generation.

Summary of findings

• A model's robotics score depends as much on the robot body and the control interface as on the model itself. The same model can look weak or strong depending on whether it is setting motor torques directly, writing a Python controller, supervising a pretrained policy, or training its own policy with reinforcement learning—each of which is a different way of the model completing the same task. For the most challenging bodies to control (the humanoid in particular), today's models only get traction at the higher-abstraction interfaces, in which a pretrained policy handles the low-level physics.
• Models are improving at robotics, but unevenly. The most consistent performance improvements between model generations are on the high-level interfaces. Direct low-level control is also improving, but much less consistently: some new models clearly improve over their predecessors, but others don’t.
• On locomotion tasks, frontier models can now perform limited but meaningful whole-body control. Newer models make progress on low level control quadruped standing, balancing, and walking—and show weaker but measurable gains on humanoid balancing. Using pretrained policies and perception tools, they can even navigate simple environments. However, models still fail at tasks that require stable spatial memory, self-localization, or long open-loop plans.
• With low-level manipulation methods, models are beginning to produce useful local physical behavior, even though full task success remains rare. Newer models are better at reaching objects, making contact, and grasping. However, they only complete the full task a small percentage of the time (from 0 to 5.5%).
• With high-level manipulation methods, newer models are more successful when using pretrained policies. Vision-language-action (VLA) scaffolds—pretrained policies that map camera images and an instruction directly to robot-arm motions—raise models’ manipulation performance far above direct control. Newer models are also becoming increasingly good supervisors of those policies: they are better at recognizing when a proposed action will fail, less likely to defer to the VLA indiscriminately, and therefore make further progress on manipulation tasks. Supervising the policy still costs some performance—the combined system does worse than the VLA running on its own—but the best supervisors now close most of the gap. That does not mean supervision is useless: earlier models destroy most of the policy's value, the best of current models recover most of it, and on tasks the VLA cannot do alone the strongest models already provide net uplift.

Simple settings

We begin by evaluating robot-relevant capabilities on a set of simple control tasks, including classic reinforcement learning (RL) problems such as balancing an inverse pendulum and controlling a hopper.

Although these are simplified environments, we believe they require the model to reason about dynamics, cause and effect over time, and basic physics—capabilities that are important precursors to more general physical understanding. In these low-dimensional environments, sensory data provides nearly all the necessary information, allowing them to be solved with little to no visual input. This kind of low-dimensional control also arises in some real-world settings, such as camera stabilization.

We evaluate models through four control interfaces, all in the simulation engine Mujoco. (Throughout, "classic control" names this family of toy tasks; "direct control" names one of the four interfaces below—they are independent axes.) In what we call direct control, the model selects low-level actions at each step, such as torques or forces. In programmatic control, the model writes a python controller that maps observations to actions during execution. In policy control, the model can access a pretrained policy and issue high-level commands, often in natural language. In reinforcement learning supervision, the model trains a policy and then deploys the learned policy at test time.

To approximate an upper bound on direct-control performance, we pause the simulator between LLM calls so real-time latency does not dominate the results. Without this, many direct tests would fail for a trivial reason: the models would simply react too slowly to control the environment. We expect inference speed to continue increasing, and this setup gives a clear view of best-case capability as it does.

We also designed our evaluation to account for the fact that many classic RL tasks appear frequently in pretraining corpora, which could limit how well our conclusions generalize to novel environments. To address this, we retain the inverse pendulum and hopper tasks as control tasks but also introduce a new task based on pinball arcade machines. In TwinFlipper, the agent controls a set of flippers and seeks to maximize the ball’s total airtime—the cumulative amount of time the ball remains above a specified height threshold while not touching anything—before it drops below the flippers. Although a naive solution is to just slam the ball upwards, much more airtime can be gained by carefully bouncing the ball up and down in a controlled manner. This task is designed to be a representative example of a chaotic, dynamic system with few degrees of freedom, and none of the models have seen it before.

While performance on individual tasks is noisy, a broader look across all classic control benchmarks shows consistent generational improvement. Claude Opus 4.6 and Opus 4.5 outperform the two earlier versions on almost all tasks except TwinFlipper-direct control, where all models perform poorly, and hopper-velocity, which is our noisiest task. Despite this, these latter two models show significant improvement in code control, reinforcement learning, and to a lesser degree direct control.

Our results suggest that much of the improvement comes from a better ability to adapt after seeing prior outcomes and to revise strategy accordingly. On tasks with a natural termination point (most notably TwinFlipper and Pendulum), average first-try performance is quite similar across models, and Claude Opus 4 and Opus 4.1 very slightly outperform later models on this measure. The larger gains appear in subsequent attempts, where later models improve much more substantially. Claude Mythos Preview is a notable exception to this; many of its first attempts are more robust in Pendulum.

Nearly all models performed worse when training an RL policy than they did when creating a Pythonic controller. The standout is TwinFlipper, where GPT-5.4 was the only model that consistently learned a competent policy. This is striking in light of its relatively poor performance under the other control interfaces. On Pendulum and Hopper the picture is different: Mythos Preview leads, GPT-5.4 is close behind, and the spread across models is much narrower. Across all three tasks, newer Claude models show a meaningful improvement over older ones.

On tasks with harder-to-specify objectives, such as Hopper and TwinFlipper, RL performance is improving but still lags behind code control. This is not surprising: having the model train its own RL policy requires solving a complex setup problem, from defining the environment and reward to managing longer iteration cycles and making several interdependent design choices. While not every Claude generation shows the same degree of improvement, the broader trend is clear: RL capabilities are advancing over time.

Direct control is bad, but improving

Low-level locomotion

The next question is whether the gains we see in simple control carry over to robots with many more degrees of freedom. To test this, we evaluated low-level locomotion on two representative platforms: the 29-DoF Unitree G1 humanoid and the 12-DoF Unitree Go2 quadruped. We note there is both a higher ceiling of contribution and greater risk profile in this domain compared to toy tasks. It is difficult to train robust humanoid and quadruped policies, but once trained they are deployed in situations in which misaligned behavior could enable serious physical harm.

These are complex robots, and controlling them numerically is challenging. Instead of controlling a few coupled variables like in simple tasks, the model must coordinate many joints while continuously compensating for gravity, inertia, and contact forces. It is very unforgiving: even tiny mistakes can destabilize the whole chassis if they are not immediately corrected.

Keeping this difficulty in mind, we evaluate the models on two core tasks: standing up from a collapsed position, and maintaining balance for as long as possible from an upright start. We initially explored more complex tasks and starting conditions, but those settings were generally beyond the reach of even frontier models. However, results on our quadruped trials were encouraging, so we also evaluated the ability to walk the Go2 robot forward programmatically.

We use three control interfaces: direct control, programmatic control, and reinforcement learning (RL). As with the classic tasks, in the direct settings, we pause the simulator between LLM calls so real-time latency does not become the limiting factor. Real-time control would require roughly 83 Hz; current non-reasoning inference runs at ~0.2-0.4 Hz, so closing this gap needs roughly two orders of magnitude latency improvement.

While direct low-level locomotive control of the quadruped is difficult for all models, many are adept at programmatic control. Opus 4.6, 4.7 and Claude Mythos Preview manage to balance the Go2 robot for nearly two full seconds, long enough to demonstrate stable balance but fast enough to iterate on, with torque-force control and a pythonic controller. Gemini 3.1 and GPT-5.4 have similarly strong controllers, although they lag far behind when controlling the motors directly. Under direct control, Opus 4.6 can keep the robot balanced but cannot successfully stand it up.

The G1 humanoid is the hardest platform in our study, and results were weak but improving. In our trials, no model successfully stood the robot up from a collapsed pose even once. Even so, there has been measurable progress between Opus 4 and 4.7 in balancing the robot once it is already in a standing position.

We also evaluated how well models can train locomotive policies. To do this, we gave them a training scaffold with access to a GPU and visualization environment, and let it control the reward function, training environment, and model architecture. Over four hours, GPT-5.4 and Claude Mythos Preview consistently train the most competent RL policies, confirming our previous results on classic RL tasks. We also observe a progression within the Claude family, with performance improving from Opus 4 to Opus 4.6 and improving further with Mythos Preview.

All of these results should be interpreted appropriately. For instance, if we randomize the initial position of the quadruped robot to include positions on its back, Opus 4.6 is unable to stand it up even once. Additionally, we reset the environment in between balancing attempts; only a few models ever achieve a robust balance on its first attempt. However, it is clear that frontier models are developing locomotive competencies.

Low-level manipulation

Manipulation is another core robotic capability with clear usefulness and safety relevance, so we study it alongside locomotion. By manipulation, we mean using a gripper or robotic arm to move and re-orient objects through a scene in a controlled way. We evaluate this capability using a fixed-base, 7-DoF Franka Panda arm in kitchen-style environments adapted from the LIBERO benchmark. These are kitchen-like tasks, such as “put the plate on the stove.”

Since the arm is anchored in place, balance is not required like it is during locomotive tasks. Instead, the challenge is controlling position, orientation, and contact precisely enough to complete the objective. A successful attempt requires the model to identify the correct object, move the arm into place, align the gripper properly, execute a stable grasp, and then transport and place the object without losing control. An error at any stage can collapse the whole attempt, although this can usually be corrected.

We test a simplified direct-control setting in which the model outputs standard seven-dimensional end-effector motion commands. After every movement, it receives images of the scene along with readings from the gripper’s force sensors, similar to what a VLA would receive. It is never given the objects’ coordinates directly, so it must first identify the relevant objects from vision and then use that information to decide how the hand should move next.

Because a fixed base arm has no real-time balance constraint, the gap between our paused simulator upper bound and real-time performance is much smaller here than for legged robots. A stationary arm under LLM control is already a plausible deployment (lab automation, light manufacturing), so even modest manipulation gains have direct safety relevance: a model that can reliably grasp, move, and reposition objects already has a meaningful ability to act on the physical world when given access to a robotic system.

The improvements are clearest in the intermediate stages of each manipulation attempt. Compared with Claude Opus 4 and Opus 4.1, Opus 4.6 is substantially more likely to guide the arm to the target object, make contact with it, and grasp it. Models still relatively rarely grasp the item, but later models tend to get further before failing and achieve higher overall task progress, as measured by a simple composite score (see Appendix for details). Interestingly, despite Claude Mythos Preview touching and grasping less, it manages to complete tasks at a significantly higher rate than the next best model, Opus 4.6, because the latter model makes more mistakes and adjustments than the former.

Despite the pace of progress between model generations, full task success is still rare; the best models cannot intentionally carry out extended tasks consistently. Even so, their ability to affect the physical world is improving in visible ways, and in rare cases they do succeed end-to-end. That level of capability may also be enough to make them useful as a source of robotic training data, and we expect future work to explore this possibility.

Tools bridge some of the gap

When models are given access to higher-level abstractions—pre-trained locomotion policies, VLAs—performance improves substantially. But the performance ceiling is still low.

High-level locomotion

To test high-level locomotion, we let the model control the quadruped robot through a pretrained joystick policy. Instead of issuing torques, it sends velocity commands (forward, lateral, yaw) to a gait policy, and periodically receives a forward-facing RGB camera frame. These policies are widely available for most commercial quadruped robots.

We built a suite of eleven navigation and spatial-reasoning tasks ranging from simple goal-seeking (find_x: walk to the table with the blue X) through search, mazes, and waypoint sequences, to tasks that explicitly probe self-monitoring (drift_detection: notice that your commands are being silently corrupted) and spatial mental-model building (explore_report: roam an arena, then answer layout questions from memory). One task, oneshot_course, removes the camera entirely and gives the model a top-down map, asking it to pre-register the entire command sequence in one shot—isolating planning from perception. Each task is scored on success or normalized progress, and we report a composite over all eleven, scaled between 0 and 100 (Appendix).

High-level locomotion improves across two clear jumps in model generation, namely between Claude Opus 4.1 to Opus 4.5 and from Opus 4.7 to Mythos Preview. Opus 4.5 through Opus 4.7 sit on somewhat of a plateau.

That plateau is an artifact of averaging: on most individual tasks, each successive Claude model moves—just not always in the same direction on every task. Comparing Opus 4.7 to Opus 4.6 at their best performing reasoning settings task-by-task, the largest single drop is on invisible_walls (3% vs 15%), where the model must replan around obstacles it cannot see. In the other direction, Opus 4.7 gains +24 points on turn_correction and +11 on return_home. We read the Opus 4.6-to-Opus 4.7 change as a shift in failure modes, like better closed-loop self-correction, and weaker replanning under occlusion.

We tested several tools to attempt to assist the model with visual and directional understanding, and more generally perception. We tried giving the model a green center crosshair drawn on its egocentric view, a semi-transparent depth heatmap alpha-blended over its view, a third-person chase camera replacing the forward view, and a "compass" which just gives the model its orientation in degrees. The compass tool handily beats the other ones, as will be elaborated upon later when we examine bottlenecks.

Takeaway: Paired with a pretrained gait, current models can complete simple navigation tasks but reliably fail tasks that require sustained spatial bookkeeping or open-loop planning. The bottleneck is primarily keeping track of where the robot is, and small bits of information can remediate some perceptual failures.

High-level manipulation

We also evaluate whether frontier models can make effective use of pretrained VLAs in manipulation settings. Our direct manipulation results show that unaided capability is still limited, even if it is improving quickly. However, a model that is only modestly capable on its own may become much more effective when paired with a pretrained policy.

To study this, we pair the model with a VLA policy on the same manipulation tasks from LIBERO. In this setting, the VLA proposes low-level actions, and the language model decides what to do with them. It can accept the proposed action, adjust it, or replace it entirely. This creates a very different kind of challenge from direct control where the core challenge is deciding which commands to accept, and which are wrong and need to be modified. We use the MolmoAct VLA across all experiments.

On the standard 40-task LIBERO benchmark, the VLA dramatically expands capability relative to direct control. Even newer models rarely complete LIBERO tasks end to end under direct control, though they can almost always make some partial progress. However, when the LLM-agent is instead allowed to guide a VLA—by giving instructions and accepting, modifying, or replacing its proposed actions—both task success and overall progress increase substantially for every model. With this augmentation, even older models achieve meaningful success rates.

We note that every tested model still performs substantially worse than MolmoAct does on its own. Counterintuitively, the penalty is not smallest for the strongest model: Claude Mythos Preview underperforms Opus 4.5 and Opus 4.6 here, because it overrides the VLA more often than is warranted, trusting its own judgment in cases where simply deferring would have succeeded. To understand where this control penalty comes from, we measure how often the agent simply follows the VLA’s proposed action. We count a command as followed only when the language model passes along the Panda arm’s full 7-dimensional action exactly as given; any edit, replacement, or omission counts as a deviation. This lets us pinpoint if the VLA’s generally sound advice is being ignored.

The results show that the Claude series of models follows the VLA instructions significantly more than GPT-5.4 and Gemini 3.1 in general. It also shows that newer models, Opus 4.5 and 4.6, follow the most instructions of any of the tested models on the LIBERO 40.

These results do not let us distinguish between a deferential model and one with good taste. To evaluate this, we tested whether these systems can use, and potentially correct, an unreliable VLA. To do so, we created three new LIBERO-like tasks drawn from the original LIBERO-goal scenes but not included in the benchmark; in our baseline trials, MolmoAct is unable to complete any of the three tasks.

Earlier Claude models as well as GPT-5.4 continue to follow the VLA relatively closely even in this setting where the VLA’s commands require corrections. By contrast, Opus 4.5, Opus 4.6 and Opus 4.7 defer to it much less. They are better at recognizing when the policy is failing, even if they are not yet able to correct those failures directly. Even so, Claude Opus 4.5 and Opus 4.6, along with Gemini 3.1, outperform MolmoAct alone. Interestingly, Opus 4 and Opus 4.1 defer to the VLA more often than Opus 4.5 and Opus 4.6 in this novel setting, yet they still achieve worse overall performance. The simplest explanation is that their higher follow rates do not reflect better judgment. They listen to the VLA at roughly the same rate they do in settings where the VLA is actually competent. Their behavior here is largely indiscriminate.

Opus 4.5, Opus 4.6, Opus 4.7, and Mythos Preview achieve the highest touch rates, grasp rates, and success on tasks the VLA is familiar with. However, only Mythos Preview is able to solve a significant portion of the novel tasks as well.

Takeaway: Pretrained policies massively boost performance: high-level control is dramatically better than low-level control. The newer Claude models are better at using pretrained policies without fighting them unnecessarily, and the models degrade less when those policies are wrong, but they still do not use the VLA as effectively as it can be used. On novel tasks, the strongest models can provide a small uplift to the high-level policy’s performance. From a safety standpoint, this matters because pretrained policies are exactly what a deployed system would realistically provide: a model does not need to drive joints itself to act capably in the world, only access to a competent controller. Capability assessments that test the model in isolation will understate what it can do once embedded in a robotic stack.

What’s the bottleneck?

Where has improvement come from in newer models, and what do they still struggle with?

Is visual perception the bottleneck?

In both manipulation and locomotion we tested whether additional visual inputs improve performance. For the Panda arm we added depth maps, labeled segmentation overlays, and a cursor tool—a small red X on the gripper cam that the model can move and query for the object and distance at that point. For the Go2 we added a depth heatmap blended over the forward camera, a green center crosshair on the image, and a third-person chase camera replacing the forward view.

On manipulation, the depth maps and segmentation overlays are roughly neutral. They convey the right kind of information, but the signal seems too diffuse to help reliably. On locomotion, the depth heatmap and the crosshair overlay are similarly close to neutral, with the depth heatmap mildly hurting the stronger models.

The third-person camera is the most model-dependent of the aids. It does nothing or slightly hurts for Opus 4.6 and earlier—Opus 4.6 drops 3.6 points—but gives Opus 4.7 +5.8 and Mythos Preview +10.7, making it Mythos Preview's single best visual aid. At the task level it helps where the model needs to track its own position over time (color_sequence, drift_detection, invisible_walls) and hurts where the task depends on the forward view, like turn_correction. Mythos Preview is the exception, improving even on turn_correction.

The cursor tool, by contrast, gives every model a large uplift on manipulation—for Mythos Preview, success on the 10-task subset goes from 6% to 32%. The compass does the same for locomotion, lifting every configuration we tested. In both cases the results suggest that models mainly need better orientation, not a different view of the scene: it is still more helpful to tell the model which way it is facing than to show it a picture of itself.

We also tested how much information an actual image provides over a detailed textual description. To do this, we replaced the visual input with text descriptions produced by the strongest image-question-answering model we tested, Gemini 3.1. In this way, we can see how well models perform when given a strong verbal description of the scene instead of pixels. If a model already uses visual input effectively, we would expect this substitution to hurt performance.

Visual perception is a more severe limitation for older models than for stronger ones. In other words, newer models are substantially better at extracting spatial information directly from images. Older Claude models perform better when images are replaced with text descriptions, which indicates that they have trouble reading enough precise spatial detail from pixels alone. Opus 4.6 and Opus 4.7, by contrast, perform slightly worse with text instead of images, and Gemini drops more as well. For them, raw visual input contains useful information that is lost when the scene is compressed into language.

Most of the additional visual inputs we tested do not help. The third-person camera helps Opus 4.7 and Mythos Preview; the other visual aids, aside from the cursors and compass, are roughly neutral or slightly negative across the board. The ask_vlm comparison shows newer models already getting more from the raw image than older ones do.

Vignettes: Vision tools on the physical Go2

Among the real-world explorations discussed later, we gave Claude Opus 4.6 control of a physical Go2 quadruped and turned on some of our visual aids during basic navigation tasks—for instance, completing a loop around the office. With the egocentric crosshair on, we could see in its reasoning that it was using the center mark to judge alignment. Walking down a hallway slightly off-axis, it noted that the hallway appeared to be drifting left of the crosshair, concluded it was probably facing too far right, and corrected. Those cases were encouraging. But the crosshair also seemed to distract from obstacles at times. In one run there was a small trash can ahead of the robot; the model recognized it and confidently stated that because the can was to the left of the crosshair, it was out of the way and safe to proceed. The trash can was in fact directly in front of the dog. It walked into it, got a leg caught, and dragged the can for a couple of meters before we stopped it.

We also tried the depth heatmap on the physical Go2. Using a computer-vision model, we overlaid estimated depth as a semi-transparent heatmap on the egocentric camera, tuned so that real-world contrast was still visible and the robot could navigate. There was some evidence the model could reason about the heatmap colors—its transcripts frequently discussed the colors in view and related them to objects being closer or in the way. But in a busier scene—a corridor turn with some plants and an office water cooler as obstacles—the model was clearly confused. It disregarded the available depth information and turned toward the obstacles instead of toward the open space.

Does reasoning help?

Reasoning had little effect on most of our results, with many differences falling within standard error.

On the classic control tasks, newer models regressed when given a higher reasoning budget. This could be due to overengineering what are relatively simple experiments. For the older models, it didn’t seem to make a big difference.

GPT-5.4, but no other model, benefited significantly from additional test-time computation during the locomotion tests. For most other models, we theorize the planning benefit that extra reasoning provides seems to also get in the way of taking nimble, reactive action.

On direct and high-level manipulation, reasoning made no major difference for any of the Claude-family models, although it affected Gemini 3.1 and GPT-5.4 significantly. Looking at the results across models, extra reasoning seems to hurt.

On high-level locomotion, reasoning budget matters very little for the Opus generations. For example, across no-reasoning, 20k-budget, and adaptive-max, Opus 4.6 lands within a 2.6-point band (37.8–40.4), and Opus 4.7 within 4.0 points. The one consistent loser is adaptive-low, which underperforms every other configuration on almost every model that supports it. Mythos Preview is the exception—its spread across configurations is nearly 14 points (40.2 at adaptive-low to 54.1 at adaptive-max), and it is the only model where additional reasoning produced a gain comparable to a perceptual aid.

We do not see strong evidence that reasoning changes how models use perceptual aids. More work is needed to understand why extra reasoning helps some models and whether it unlocks abilities that are already latent.

These findings suggest that additional reasoning alone, in current generation models, is unlikely to overcome the deficiencies that currently prevent models from performing general low-level robotics. While some models benefit from the additional reasoning, the leap in capabilities between generations is made up of other skills, like better vision, numerical consistency, or 3D understanding.

Can they learn from experience?

Yes—but mostly over short horizons.

While it is well-known that language models perform better under few-shot settings, this does not mean that they can learn from long-context robotic embodiment settings, which can span hundreds of images and hundreds of thousands of tokens.

The strongest evidence for in-context learning comes from classic control tasks. Later Claude models do not pull ahead by starting much stronger—nearly all first attempts perform poorly, with only a few exceptions. Instead, they improve by in-context learning from failed attempts and performing better control. Opus 4.5 and Opus 4.6 benefit much more from iteration than Opus 4 and Opus 4.1. Newer versions of Claude are better able to learn from failed attempts, revise their approach, and find working solutions.

Long-horizon robotic interaction requires more than just choosing the next action correctly. Although the tasks we study are, in principle, mostly Markovian, successful performance still unfolds over hundreds or more precise commands. In practice, the system uses this extended interaction to learn how the task behaves and filter out ineffective tactics.

We conduct a test to learn whether models can build up a richer, longer-range understanding over the course of a trial. We examine this with context-truncation experiments in manipulation, where we deliberately remove most of the prior interaction and leave the model only a much smaller window of recent actions and observations. If performance depended on a detailed memory of the full episode, this should have caused a large drop. In most cases, it did not. In some cases, performance even improved. This indicates that the models rely much more on the recent past than on a broad accumulated understanding of everything that happened earlier.

In these truncation runs we always retain the first 10 turns plus the most recent N turns. Dropping the first turn caused models to forget basic conventions and loop, so we keep it in all conditions.

Only Opus 4.6 showed a statistically significant drop in performance. It is notable that Claude Mythos Preview, which was generally the strongest model in our tests, did not experience significantly degraded accuracy. We suspect that Opus 4.6 is continually learning behavior patterns that it has forgotten because of the dropped context, whereas Claude Mythos Preview can utilize these strategies out-of-the-box. Weaker models may get confused by earlier context in a documented phenomenon known as “context rot,” which explains why performance increases when context is truncated. These models were unable to learn from the distant past, so its removal is a performance enhancer.

We saw earlier that newer Claude models are more likely to change strategy after failure, and this drives part of their performance gain. The context-truncation results indicate that this kind of adaptation is mostly short-range. Models do reframe and adjust, but they appear to do so mainly based on the most recent few steps and do not need to develop a long-running strategy built up over a full episode. In particular, actions from much earlier in the interaction do not seem to matter much and removing them changes performance very little.

We also see evidence for short-term learning in high-level locomotion. In the oneshot_course task, the model has to look at a simple top-down map of an L-shaped hallway and plan the full set of movement commands before it starts. There is no camera input and no chance to adjust along the way. Without practice, models generally struggle, which suggests that the task is not solved by basic map reading alone. It requires turning the map into an action plan that works on the first try.

When we give the models a few practice runs on the same course, performance improves across the board. The main difference in performance is how quickly they learn. Mythos Preview reaches strong performance after just one example, while Opus improves more gradually over several tries. Even smaller models can learn the course with enough practice. Overall, this suggests that the ability to learn from a few in-context examples is broadly present, but Mythos Preview stands out because it needs far less practice to use that information well.

On harder, longer courses, practice does not help. On those trials, neither Mythos Preview nor Opus 4.7 completes a single trial even with twenty practice runs. The models are learning a specific sequence and are not yet general planners.

Real-world vignettes

The vision-tool vignettes above already used a physical Unitree Go2. This section reports the rest of our real-world runs on that robot. Although we could not achieve high N count trials due to the serial nature of work in the real world, our explorations generally aligned with our simulated findings, but elucidated some interesting failure cases that all centered around poor visual and spatial reasoning.

Firstly, we were able to reimplement the find_x task. In this task, the quadruped was placed ~25 feet and facing 180 degrees away from an overturned table with a large blue X on it. The models were instructed to find the table and walk all the way up to it (within at least a meter distance). The most common failure case, that was also found in simulation, was that many models would simply stop short of the 1 meter distance required to succeed at the task. Additionally, older models would fail to correct course on their way to the target object. The common behavior across all models for this task is that models would rotate until they saw the table in frame, and proceed to the table. Older models would not accurately align themselves and miss the table, often convincing themselves that they’re on path, or failing to realize that the table was sharply to one side or the other and reaching a point where they're getting further from the table while claiming to get nearer to it. One failure in the find_x real world replication is the testing of Grok 4.1 Fast. In this case, the Go2 was positioned in such a way that it was facing a glass door opposite to the table; Grok saw the target table in the reflection of the glass door and started charging for the glass door. Thankfully, the robot was stopped before any damage was incurred to either the door or itself.

Furthermore, an informal benchmark was to ask a model to control the Go2 and have it complete one loop around the office hallway circuit (with just vision alone). Regardless of the various harnesses and models we used and advantages we tried to give the model, all models failed at this task. This failure mode is predominantly caused by vision and memory failures. Sometimes a model simply cannot tell when it is time to turn as it passes by the opening to another corridor. Other times, it thinks it has turned into the corridor and thinks it has walked well into it when in fact it hasn't; it attempts to turn again or go in the wrong direction. And even if a model can turn a corridor, sometimes a model overshoots or undershoots a turn, gets confused, and usually ends up going in the opposite direction.

Conclusion

Our experiment suite shows rapid, if unequal, improvement in robotics tasks across model generations. Newer Claude models are better at turning perception and reasoning into physical action across a host of embodiments. Direct force and torque control is improving, but more slowly than higher-level control.

This research has clear safety implications. A VLM’s real-world influence can change by orders of magnitude depending on the information it has access to. Evaluations and deployments need to treat access level as a core part of the system, because small changes in tools or control can produce large changes in capability.

We hope these results guide work in both directions. On the constructive side, models may help robots debug failures, supervise existing controllers, and generate useful training data. On the safety side, we need better ways to grant physical access with clear limits, so a system can affect certain objects while being blocked from others.

Appendix

This appendix summarizes the practical details behind the evaluations: which model APIs we used, how many trials we ran, how prompts were structured, how latency affected the setup, and how the reinforcement learning runs worked.

Models and APIs

We evaluated twelve models across five providers, with four of them through OpenRouter. To keep the evaluation harness the same across providers, we wrote a small adapter for each backend. The adapter handled model-specific API calls, but the robot tasks, prompts, and scoring code stayed the same.

A few implementation details matter for reproducibility:

• Claude models. Claude models ran through the Anthropic Agent SDK. During evaluation, we disabled the SDK’s built-in tools and exposed only our robot action server. This kept Claude’s available actions aligned with the OpenRouter and Gemini runs.
• Reasoning settings. Newer Claude models used Anthropic’s adaptive reasoning setting, where the model chooses how much reasoning budget to spend. Older Claude models used fixed reasoning budgets. All other models use the high reasoning setting equivalent.
• OpenRouter settings. Some OpenRouter models do not support a true “no reasoning” mode, so we used the closest available setting for those vendors. Kimi K2.6 was pinned to the Novita provider because other routes used lower-precision versions. For OpenAI models, we excluded the Azure provider because it silently capped requests at 50 images.

Trial counts

A “cell” means one model evaluated on one experiment setting. Most cells used 35 trials, but some settings used more when the task was noisier or when we needed tighter estimates.

Prompts

We did not tune prompts separately for each model. Each interface used one fixed prompt template. At trial time, the template was filled with task-specific details such as joint count, robot mass, force limits, and observation fields.

Code control

In code control, the model writes a Python controller, usually a function of the form controller(obs) -> action, and then runs it.

VLA-supervised manipulation

In this setting, a pretrained vision-language-action policy proposes robot-arm actions, and the language model decides whether to accept, edit, or replace them.

Reinforcement learning supervision

In this setting, the model writes the reward function, policy network, and training schedule. It then trains a policy and deploys it.

Latency

We did not run a formal latency study. The experiments used shared infrastructure, and API latency varied by provider, load, image count, and reasoning budget. Still, the launch logs give a useful rough picture.

• Without reasoning, most text-only turns took about 2–8 seconds. With one or two images, this usually rose to 5–15 seconds.
• With reasoning, latency depended heavily on the reasoning budget. For Opus 4.6 and 4.7 at high reasoning, typical turns took 15–60 seconds, with longer tails of 60–180 seconds. Extra-high reasoning was about twice as slow on average.
• Full cell runtime varied widely. A 35-trial direct or code cell usually took 30–90 minutes without reasoning and 1–4 hours with reasoning. LIBERO-40 direct runs took 6–12 hours per model and condition. LIBERO-40 VLA+LLM runs took 8–18 hours because they also required GPU inference for MolmoAct. RL cells were capped at 1.5 hours for classic control and 4 hours for G1 and Go2.

For direct-control and code-control simulations, we paused the simulator while the model produced its next action. Without pausing, current API models would fail for a trivial reason: they act far too slowly for a physics loop running at 10–125 Hz. Pausing lets us measure what the model could do if inference were faster, rather than only measuring today’s API latency.

The robot arm did not require the same tight real-time stability as locomotion, so we did not pause the simulator between model calls in the manipulation setting.

Reinforcement learning details

The reinforcement learning interface used the live PPO training path in envapi/training_bridge.py:train_ppo_batched. PPO is a standard reinforcement learning algorithm.

The model was allowed to define the reward function, policy network, and training schedule. It could then call training and deployment tools.

Environment

We used a GPU-backed batched MuJoCo environment called BatchedEnvWarp. This let the model train over several simulation copies in parallel.

Default training settings

The model could change these settings within guardrails, but the defaults were:

Limits and safeguards

• The harness exposed up to 32 parallel environments by default.
• The policy network was capped at 200,000 parameters.
• A single training call could not consume more than one third of the session.
• The system reserved the final 120 seconds for policy deployment.

RL cells

The RL path was used for:

• Pendulum RL
• Hopper RL
• TwinFlipper RL
• G1 Stand RL
• Go2 Stand-from-prone RL

Classic-control RL cells had a 1.5-hour timeout. Humanoid and quadruped RL cells had a 4-hour timeout.

Vision inputs

When vision was enabled, the harness sent JPEG-encoded RGB frames to the model.

Image format

Frames were rendered off-screen in MuJoCo and encoded as JPEGs. Each provider received images in its native format:

• Anthropic image blocks
• OpenAI-style image_url with data URIs
• Gemini inline image parts

Frames per turn

The default harness setting allows three frames per turn, but all published cells used one frame per turn.

Keeping prior frames

All published vision cells kept prior frames in context. This was required for Claude Agent SDK runs with vision, because that SDK path did not support deleting older images turn by turn.

Vision tools

We also tested several ways of changing what visual information the model received:

• --vision-depth: adds a depth heatmap overlay
• --tools segmentation: adds a labeled segmentation map
• --crosshair --gripper-cam: adds an interactive cursor (formerly called crosshair) on the gripper camera
• --vision-mode ask_vlm: replaces pixels with natural-language scene descriptions from Gemini 3.1 Pro Preview

For the high-level locomotion suite, we tested five perceptual-aid conditions in addition to the baseline forward camera:

• compass: the robot's world-frame heading in degrees, appended as text alongside each frame
• crosshair: a green center crosshair drawn on the forward camera image
• depth: a semi-transparent depth heatmap alpha-blended over the forward camera image
• third_person: the forward camera is replaced with a third-person chase camera positioned behind and above the robot
• combo: all four aids applied together

The “ask VLM” setting added latency because every visual query required an extra model call. It also reduced performance for the strongest models, which indicates that those models were using information from the raw images that was lost when the scene was converted into text.

Compute setup

All experiments ran on a cluster managed by SLURM, a common system for scheduling large compute jobs.

• Direct-control and code-control cells used CPU-only nodes, usually with 16 CPUs, 48 GB of RAM, and a 24-hour wall-time limit.
• VLA cells used one GPU per job for MolmoAct inference, plus CPU rendering for the simulator.
• RL cells used one GPU per job. Classic-control RL runs had 1.5-hour sessions. G1 and Go2 RL runs had 4-hour sessions.
• Rendering used off-screen MuJoCo through osmesa, with MUJOCO_GL=osmesa and PYOPENGL_PLATFORM=osmesa.

All scripts disabled legacy Claude model remapping, loaded the project virtual environment, loaded the environment variables, checked that libosmesa6 was installed, and passed --no-sdk-tool-only for Claude SDK runs.

Reproducibility

The code, once released, will be in github.com/safety-research/embody, the public mirror of the repository. The command for each evaluation cell is listed in EXPERIMENTS.md, and scoring is documented in METRICS.md.