Gemini 4 Argon for Robotics

[Attached: wrist_cam.png, obs.json]

Task: Get the robot arm to reorient the mug upright and put the napkin inside the mug.

Execution Discipline

• Simply output 1 sentence observation followed immediately by your curl command (with # Plan: on Turn 1 and # Thought: on every turn). Everything needed is in obs.json and the calibrated reticle/grid drawn on wrist_cam.png.

Visual Coordinate Reference (Annotated on wrist_cam.png)

• +X (UP in image): Away from robot base | -X (DOWN): Towards robot base
• +Y (LEFT in image): World left | -Y (RIGHT): World right
• +Z: Height above table | Yellow numbers: World (X, Y) in centimetres (2 cm grid at grid_z)
• Cyan + crosshair: Current 3D pinch center
• Cyan [ + ] brackets: Current jaw opening and closing line
• Green line: Current object long-axis orientation (yaw_deg: 0° = vertical along +X, 90° = horizontal along +Y)
• pitch_deg = 90°: Straight down (-Z) top-down grasp/place; pitch_deg < 60°: Tilted coaxial approach along finger axis

Closed-Loop Turn Workflow

1. Inspect current state & view wrist_cam.png: curl -s http://127.0.0.1:8081/obs (returns ee_pos, holding, status, workspace, wrist_cam_path).
2. Select VLM call granularity upfront on Turn 1:

• Coarse (2–4 calls): look_at -> grasp -> place -> done
• Pre-Grasp Hover Verify (4–7 calls): look_at -> goto(z=0.04) (verify cyan [ + ] brackets straddle target) -> grasp -> place_until_contact -> done
• Fine Visual Servoing (8–20 calls): look_at -> grasp -> pre-pose goto -> closed-loop move_by / move_until_contact -> done

3. Send motion primitives (POST /plan) and verify visually after every turn in the updated wrist_cam.png. If status == "grasp_failed", call robot.look_at(...) to re-observe and recover closed-loop.

Code-as-Policies Primitives (POST http://127.0.0.1:8081/plan)

robot.look_at(x=0.35, y=0.13, z=0.14, grid_z=0.015) x, y = robot.deproject(u=0.52, v=0.48, z_plane=0.015) robot.grasp(x=x, y=y, z=0.015, yaw_deg=-35, pitch_deg=90) robot.place(x=0.24, y=0.06, z=0.075) robot.place_until_contact(x=0.24, y=0.06, z=0.015, max_lag=0.012, relief=0.002) robot.goto(x=0.28, y=0.05, z=0.04, arc_z=0.14, pitch_deg=25, azimuth_deg=90, roll_deg=0, speed=0.10) robot.move_by(forward=0.025, dz=-0.002, dpitch_deg=-3, droll_deg=5, speed=0.04) robot.move_until_contact(dz=-0.06, speed=0.03, max_lag=0.012, relief=0.002) robot.open_gripper() / robot.close_gripper() / robot.home() / robot.done()

Plain text

Copied

3. Omni control

Argon steers robots in the Gemini Omni World Model to rearrange five scattered Scrabble tiles (G, R, A, N, O) into a clean row spelling A - R - G - O - N. Instead of specifying one static prompt, Argon steers in a closed visual loop. It writes timestamped action prompts, inspecting intermediate rollout keyframes, and refining its prompts.

Sorry, your browser doesn't support playback for this video