
AA-Robotics v0.1
We are launching AA-Robotics v0.1, our research preview that measures frontier language model performance on understanding, planning, and acting in the physical world. In v0.1, each model is given control of a 6-axis ro…
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
Artificial AnalysisK
Artificial AnalysisModels
Coding Agents
Image, Speech, Video
Inference
Leaderboards
About
AI Trends
K
All articles
We are launching AA-Robotics v0.1, our research preview that measures frontier language model performance on understanding, planning, and acting in the physical world. In v0.1, each model is given control of a 6-axis robot arm (5 joints and a gripper) and a simple objective. We’re releasing results for our first eight tasks today.
Introducing AA-Robotics v0.1The eight tasksResultsHow AA-Robotics worksTime & costHow each model spent its timePhysical accuracyMistake wall
Introducing AA-Robotics v0.1
Since the beginning of 2026, frontier language models have made enormous progress in their ability to control robot arms. We now believe that the future of robotics is likely to involve strong generalist agents, and conclude that tracking progress of language models for robotics tasks is going to be important - even if frontier language models are unsuitable for production robotics use on their own.
AA-Robotics v0.1 is our research preview of the simplest experiment we have designed to test this - language models in coding agent harnesses directly controlling a robot arm. Each model is given control of a 6-axis robot arm (5 joints and a gripper) and a simple objective for each task.
We find that frontier language models in coding agent harnesses have indeed recently become capable of directly manipulating robot arms for simple tasks. Today, they are far too slow and expensive for any kind of production usage - nearly two orders of magnitude worse than human baselines. We expect this to change quickly.
GPT-6 Astra places first in AA-Robotics v0.1 with a score of 69, followed by GPT-6.1 Sol, Gemini 4 Argon, Claude Opus 5.5 and Claude Fable 5.1. GPT-6 Astra took 30 minutes and cost $16.36 on average per task.
Each attempt is scored out of 100: completing the task scores 100, and a partial result earns points for the progress made.
AA-Robotics v0.1 initially includes eight tabletop tasks, modeled on everyday activities, to evaluate a range of robotics capabilities: control and dexterity, such as precise grasping and handling objects, spatial reasoning, such as understanding how objects interact with one another, and adaptability, measuring how models respond to the unexpected.
Full score (100)
Partial (1 to 99)
Zero (0)
TaskGPT-6 Astra
GPT-6.1 Sol
Gemini 4 Argon
Claude Opus 5.5
Claude Fable 5.1
Press the red buttonFull score, 100 of 100
Full score, 100 of 100
Full score, 100 of 100
Full score, 100 of 100
Full score, 100 of 100
Place the crocodile toy in the rectangleFull score, 100 of 100
Full score, 100 of 100
Full score, 100 of 100
Full score, 100 of 100
Full score, 100 of 100
Pour the drink into the cupFull score, 100 of 100
Zero, 0 of 100
Full score, 100 of 100
Partial, 25 of 100
Partial, 30 of 100
Tidy the blocks into the boxPartial, 40 of 100
Full score, 100 of 100
Partial, 40 of 100
Partial, 40 of 100
Partial, 40 of 100
Complete the shape matching challengePartial, 44 of 100
Full score, 100 of 100
Zero, 0 of 100
Partial, 38 of 100
Full score, 100 of 100
Write hello worldFull score, 100 of 100
Full score, 100 of 100
Partial, 45 of 100
Partial, 45 of 100
Zero, 0 of 100
Build the tallest towerPartial, 71 of 100
Zero, 0 of 100
Partial, 57 of 100
Partial, 25 of 100
Zero, 0 of 100
Measure the boxZero, 0 of 100
Partial, 17 of 100
Zero, 0 of 100
Partial, 17 of 100
Zero, 0 of 100
AA-Robotics Score69.464.655.348.846.3
Each attempt is scored out of 100: completing the task scores 100, and a partial result earns points for the progress made.
GPT models run in Codex, Claude models in Claude Code, and Gemini 4 Argon in Antigravity, all at the high effort setting.
0:00 / 0:00Movement only8x16x32x64x
How AA-Robotics works
TaskAttemptAttempt EndResult ValidatedPlain-text instructionPour the drinkfrom the bottle intothe cup.Language modelCamerasRobot armThe model endsthe attempt whenit judges the taskcomplete, or gives upA human judgescores the end state
Models run in an isolated container with brief instructions: how to reach the robot arm (via a simple HTTP API), which arm it is (a Hiwonder NexArm, with no further specifications), and how to submit an attempt as complete. They can view the scene through four cameras, three mounted around the table and one on the arm’s wrist, and can write and run code in the container to plan and send movements.
Once a model has submitted a task for review, the end state is graded against the task’s pass criteria. A pass scores 100, with points awarded for partial success in the case of an incomplete task. All task attempts are zero-shot.
Today’s language models are roughly 100 times slower than a human on AA-Robotics v0.1, taking about half an hour per task compared with a human average of 20 seconds. In aggregate, four of the five models tested look similar, at about 30 minutes per task, while Gemini 4 Argon is the slowest at 52 minutes. However, these averages hide large differences between tasks.
GPT-6 Astra pressed the red button in 2.2 minutes, whereas Claude Fable 5.1 spent about ten minutes understanding and planning before its first action. The largest difference came on the crocodile placement task, where GPT-6.1 Sol declared itself complete after five minutes, compared with 45 for Claude Opus 5.5, a ninefold difference. Across all tasks, Gemini 4 Argon was the slowest to act, spending an average of 21 minutes observing and reasoning before its first action, nearly twice as long as any other model.
AA-Robotics Score vs. Time per Task
Average time per task (minutes) · All tasks
Most attractive quadrant
Model Pareto frontier
The gap between humans and models is narrower on cost: the most cost-effective model costs about ten times as much as a person paid the average US hourly wage, against nearly 100 times as long on time. Between models, GPT-6.1 Sol is the cheapest, at $1.93 per task on average, while GPT-6 Astra, the most expensive, costs $16.36, more than eight times as much as Sol, for a score only four points higher (69 against 65). Given the novelty of the arm and scene, and the large volume of visual information the models must process, most of the cost lands on input: 70% of the spend goes on context.
OutputCached InputInputHuman
AA-Robotics Score vs. Cost per Task
Average cost per task (USD) · All tasks
Most attractive quadrant
Model Pareto frontier
Human cost is the by-hand task time priced at $37.81 per hour, the average hourly earnings of all employees on US private nonfarm payrolls (US Bureau of Labor Statistics, preliminary data for September 2026).
As expected, models spend a large portion of the initial task time calibrating the arm and understanding the scene, continuing this up until taking the first action. After this point, the majority of time is spent planning movement, with relatively short, precise arm movements between each longer planning step. Camera views and note writing are relatively small fractions of the overall time.
Interestingly, models often continue to refine their attempts long after reaching a passing state. GPT-6.1 Sol, for example, builds a passing tower after ~17 minutes, only to subsequently fail the task. It remains to be seen how models will improve their ability to reason about task completion in physical space.
Understanding the Scene, Arm, and Movement
Planning Movements
Reviewing Cameras
Arm Moving
Other
Gemini 4 Argon42%
45%
6%
6%
419 min
Claude Fable 5.133%
37%
9%
18%
249 min
Claude Opus 5.521%
45%
9%
23%
253 min
GPT-6 Astra20%
59%
6%
13%
242 min
GPT-6.1 Sol17%
61%
6%
13%
224 min
Other covers waiting on running code, writing notes, and the final report.
Each step’s thinking and tool time is assigned to one activity from its code, images, and messages; arm moving is the time the arm was carrying out a command.
Gemini 4 Argon's activity times are estimated: Antigravity does not timestamp each step, so steps are placed between the arm commands, which are timed exactly, in proportion to their tokens.
First Reached a Passing State
Irrecoverable Mistake
0 min5 min10 min15 min20 min25 min30 min35 min40 minGemini 4 ArgonFirst action at 25:36First reached a passing state at 32:15Claude Fable 5.1Irrecoverable mistake at 14:08First action at 7:51Claude Opus 5.5Irrecoverable mistake at 20:59First action at 4:14GPT-6 AstraFirst action at 2:23First reached a passing state at 7:32GPT-6.1 SolIrrecoverable mistake at 5:27First action at 2:39
Other covers waiting on running code, writing notes, and the final report.
The first action is the first press, grasp, push, pour, or pen stroke of a run.
An irrecoverable mistake means the model can no longer achieve full marks.
Gemini 4 Argon's activity times are estimated: Antigravity does not timestamp each step, so steps are placed between the arm commands, which are timed exactly, in proportion to their tokens.
Grasping and fine manipulation were key skills to test in AA-Robotics v0.1, as they point the way not only to more precise control tasks, but more importantly to tool use and the far wider range of capabilities it unlocks.
GPT-6 Astra and Claude Fable 5.1 performed best with the highest proportion of successful attempts (~45%), indicating a degree of close control, whilst GPT-6.1 Sol, scored worst, at 28%. Gemini 4 Argon fell in between at 36%, with many of its misses closing just above or beside the block it was reaching for.
Successful Grasp
Missed, Partial, or Wrong-Object Grasp
GPT-6 Astra45%
55%
n = 95
Claude Fable 5.144%
56%
n = 89
Claude Opus 5.540%
60%
n = 100
Gemini 4 Argon36%
64%
n = 75
GPT-6.1 Sol28%
72%
n = 111
Grasps were attempted in the Place Crocodile, Pour Drink, Tidy Box, Match Shapes, Build Tower, and Measure Box tasks. Successful is defined as holding the intended object through its lift or carry, and missed, partial, or wrong-object where the gripper did not close on the object, closed on it before it slipped, dropped, or could not be lifted, or took a different object by mistake.
Hover over a clip to see what went wrong and click a clip to enlarge it. We’ve highlighted a few of our favorite examples!
Misjudging success or failure - and when to stop
Has the crocodile inside the rectangle at 23:22, keeps adjusting for another 22 minutes, and pushes it back over the far edge three times
Has the crocodile inside the rectangle by 24:34, then spends seven more minutes repositioning it for margin
Rewrites the phrase over the same patch of paper at least six more times after an illegible first attempt
Judges the off-center lid flush and reports the task complete
Poor physical reasoning
Misjudges where the leaning block's edge is and lifts it by the corner, without anticipating that this will knock over the tower
Missed grasps, misplacements and drops
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。