返回资讯中心
外部精选
软件工程
#开发者实时流

AA-Robotics v0.1

We are launching AA-Robotics v0.1, our research preview that measures frontier language model performance on understanding, planning, and acting in the physical world. In v0.1, each model is given control of a 6-axis ro…

Hacker News 最新liamswayne2026/10/15 · 2026/10/1527 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

Artificial AnalysisK

Artificial AnalysisModels

Coding Agents

Image, Speech, Video

Inference

Leaderboards

About

AI Trends

K

All articles

We are launching AA-Robotics v0.1, our research preview that measures frontier language model performance on understanding, planning, and acting in the physical world. In v0.1, each model is given control of a 6-axis robot arm (5 joints and a gripper) and a simple objective. We’re releasing results for our first eight tasks today.

Introducing AA-Robotics v0.1The eight tasksResultsHow AA-Robotics worksTime & costHow each model spent its timePhysical accuracyMistake wall

Introducing AA-Robotics v0.1

Since the beginning of 2026, frontier language models have made enormous progress in their ability to control robot arms. We now believe that the future of robotics is likely to involve strong generalist agents, and conclude that tracking progress of language models for robotics tasks is going to be important - even if frontier language models are unsuitable for production robotics use on their own.

AA-Robotics v0.1 is our research preview of the simplest experiment we have designed to test this - language models in coding agent harnesses directly controlling a robot arm. Each model is given control of a 6-axis robot arm (5 joints and a gripper) and a simple objective for each task.

We find that frontier language models in coding agent harnesses have indeed recently become capable of directly manipulating robot arms for simple tasks. Today, they are far too slow and expensive for any kind of production usage - nearly two orders of magnitude worse than human baselines. We expect this to change quickly.

GPT-6 Astra places first in AA-Robotics v0.1 with a score of 69, followed by GPT-6.1 Sol, Gemini 4 Argon, Claude Opus 5.5 and Claude Fable 5.1. GPT-6 Astra took 30 minutes and cost $16.36 on average per task.

Each attempt is scored out of 100: completing the task scores 100, and a partial result earns points for the progress made.

AA-Robotics v0.1 initially includes eight tabletop tasks, modeled on everyday activities, to evaluate a range of robotics capabilities: control and dexterity, such as precise grasping and handling objects, spatial reasoning, such as understanding how objects interact with one another, and adaptability, measuring how models respond to the unexpected.

Full score (100)

Partial (1 to 99)

Zero (0)

TaskGPT-6 Astra

GPT-6.1 Sol

Gemini 4 Argon

Claude Opus 5.5

Claude Fable 5.1

Press the red buttonFull score, 100 of 100

Full score, 100 of 100

Full score, 100 of 100

Full score, 100 of 100

Full score, 100 of 100

Place the crocodile toy in the rectangleFull score, 100 of 100

Full score, 100 of 100

Full score, 100 of 100

Full score, 100 of 100

Full score, 100 of 100

Pour the drink into the cupFull score, 100 of 100

Zero, 0 of 100

Full score, 100 of 100

Partial, 25 of 100

Partial, 30 of 100

Tidy the blocks into the boxPartial, 40 of 100

Full score, 100 of 100

Partial, 40 of 100

Partial, 40 of 100

Partial, 40 of 100

Complete the shape matching challengePartial, 44 of 100

Full score, 100 of 100

Zero, 0 of 100

Partial, 38 of 100

Full score, 100 of 100

Write hello worldFull score, 100 of 100

Full score, 100 of 100

Partial, 45 of 100

Partial, 45 of 100

Zero, 0 of 100

Build the tallest towerPartial, 71 of 100

Zero, 0 of 100

Partial, 57 of 100

Partial, 25 of 100

Zero, 0 of 100

Measure the boxZero, 0 of 100

Partial, 17 of 100

Zero, 0 of 100

Partial, 17 of 100

Zero, 0 of 100

AA-Robotics Score69.464.655.348.846.3

Each attempt is scored out of 100: completing the task scores 100, and a partial result earns points for the progress made.

GPT models run in Codex, Claude models in Claude Code, and Gemini 4 Argon in Antigravity, all at the high effort setting.

0:00 / 0:00Movement only8x16x32x64x

How AA-Robotics works

TaskAttemptAttempt EndResult ValidatedPlain-text instructionPour the drinkfrom the bottle intothe cup.Language modelCamerasRobot armThe model endsthe attempt whenit judges the taskcomplete, or gives upA human judgescores the end state

Models run in an isolated container with brief instructions: how to reach the robot arm (via a simple HTTP API), which arm it is (a Hiwonder NexArm, with no further specifications), and how to submit an attempt as complete. They can view the scene through four cameras, three mounted around the table and one on the arm’s wrist, and can write and run code in the container to plan and send movements.

Once a model has submitted a task for review, the end state is graded against the task’s pass criteria. A pass scores 100, with points awarded for partial success in the case of an incomplete task. All task attempts are zero-shot.

Today’s language models are roughly 100 times slower than a human on AA-Robotics v0.1, taking about half an hour per task compared with a human average of 20 seconds. In aggregate, four of the five models tested look similar, at about 30 minutes per task, while Gemini 4 Argon is the slowest at 52 minutes. However, these averages hide large differences between tasks.

GPT-6 Astra pressed the red button in 2.2 minutes, whereas Claude Fable 5.1 spent about ten minutes understanding and planning before its first action. The largest difference came on the crocodile placement task, where GPT-6.1 Sol declared itself complete after five minutes, compared with 45 for Claude Opus 5.5, a ninefold difference. Across all tasks, Gemini 4 Argon was the slowest to act, spending an average of 21 minutes observing and reasoning before its first action, nearly twice as long as any other model.

AA-Robotics Score vs. Time per Task

Average time per task (minutes) · All tasks

Most attractive quadrant

Model Pareto frontier

The gap between humans and models is narrower on cost: the most cost-effective model costs about ten times as much as a person paid the average US hourly wage, against nearly 100 times as long on time. Between models, GPT-6.1 Sol is the cheapest, at $1.93 per task on average, while GPT-6 Astra, the most expensive, costs $16.36, more than eight times as much as Sol, for a score only four points higher (69 against 65). Given the novelty of the arm and scene, and the large volume of visual information the models must process, most of the cost lands on input: 70% of the spend goes on context.

OutputCached InputInputHuman

AA-Robotics Score vs. Cost per Task

Average cost per task (USD) · All tasks

Most attractive quadrant

Model Pareto frontier

Human cost is the by-hand task time priced at $37.81 per hour, the average hourly earnings of all employees on US private nonfarm payrolls (US Bureau of Labor Statistics, preliminary data for September 2026).

As expected, models spend a large portion of the initial task time calibrating the arm and understanding the scene, continuing this up until taking the first action. After this point, the majority of time is spent planning movement, with relatively short, precise arm movements between each longer planning step. Camera views and note writing are relatively small fractions of the overall time.

Interestingly, models often continue to refine their attempts long after reaching a passing state. GPT-6.1 Sol, for example, builds a passing tower after ~17 minutes, only to subsequently fail the task. It remains to be seen how models will improve their ability to reason about task completion in physical space.

Understanding the Scene, Arm, and Movement

Planning Movements

Reviewing Cameras

Arm Moving

Other

Gemini 4 Argon42%

45%

6%

6%

419 min

Claude Fable 5.133%

37%

9%

18%

249 min

Claude Opus 5.521%

45%

9%

23%

253 min

GPT-6 Astra20%

59%

6%

13%

242 min

GPT-6.1 Sol17%

61%

6%

13%

224 min

Other covers waiting on running code, writing notes, and the final report.

Each step’s thinking and tool time is assigned to one activity from its code, images, and messages; arm moving is the time the arm was carrying out a command.

Gemini 4 Argon's activity times are estimated: Antigravity does not timestamp each step, so steps are placed between the arm commands, which are timed exactly, in proportion to their tokens.

First Reached a Passing State

Irrecoverable Mistake

0 min5 min10 min15 min20 min25 min30 min35 min40 minGemini 4 ArgonFirst action at 25:36First reached a passing state at 32:15Claude Fable 5.1Irrecoverable mistake at 14:08First action at 7:51Claude Opus 5.5Irrecoverable mistake at 20:59First action at 4:14GPT-6 AstraFirst action at 2:23First reached a passing state at 7:32GPT-6.1 SolIrrecoverable mistake at 5:27First action at 2:39

Other covers waiting on running code, writing notes, and the final report.

The first action is the first press, grasp, push, pour, or pen stroke of a run.

An irrecoverable mistake means the model can no longer achieve full marks.

Gemini 4 Argon's activity times are estimated: Antigravity does not timestamp each step, so steps are placed between the arm commands, which are timed exactly, in proportion to their tokens.

Grasping and fine manipulation were key skills to test in AA-Robotics v0.1, as they point the way not only to more precise control tasks, but more importantly to tool use and the far wider range of capabilities it unlocks.

GPT-6 Astra and Claude Fable 5.1 performed best with the highest proportion of successful attempts (~45%), indicating a degree of close control, whilst GPT-6.1 Sol, scored worst, at 28%. Gemini 4 Argon fell in between at 36%, with many of its misses closing just above or beside the block it was reaching for.

Successful Grasp

Missed, Partial, or Wrong-Object Grasp

GPT-6 Astra45%

55%

n = 95

Claude Fable 5.144%

56%

n = 89

Claude Opus 5.540%

60%

n = 100

Gemini 4 Argon36%

64%

n = 75

GPT-6.1 Sol28%

72%

n = 111

Grasps were attempted in the Place Crocodile, Pour Drink, Tidy Box, Match Shapes, Build Tower, and Measure Box tasks. Successful is defined as holding the intended object through its lift or carry, and missed, partial, or wrong-object where the gripper did not close on the object, closed on it before it slipped, dropped, or could not be lifted, or took a different object by mistake.

Hover over a clip to see what went wrong and click a clip to enlarge it. We’ve highlighted a few of our favorite examples!

Misjudging success or failure - and when to stop

Has the crocodile inside the rectangle at 23:22, keeps adjusting for another 22 minutes, and pushes it back over the far edge three times

Has the crocodile inside the rectangle by 24:34, then spends seven more minutes repositioning it for margin

Rewrites the phrase over the same patch of paper at least six more times after an illegible first attempt

Judges the off-center lid flush and reports the task complete

Poor physical reasoning

Misjudges where the leaning block's edge is and lifts it by the corner, without anticipating that this will knock over the tower

Missed grasps, misplacements and drops

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →