REKI – Registering Knowledge in VLAs
Juliette Marrie1, Paul Pacaud2, Adrien Ramanana
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
Juliette Marrie1, Paul Pacaud2, Adrien Ramanana Rahary1,3, Shizhe Chen2, and Moritz Böhle1
We introduce REKI, a 2.4B vision-language-action model (VLA) pretrained exclusively on public data that sets a new Pareto frontier between performance and model size: on LIBERO-Plus, RoboTwin Clean2Rand and RoboCasa365, it is outperformed only by models about twice its size. REKI consists of a vision-language model (VLM) followed by a flow-matching action expert, and is pretrained on a curated mixture of public robot, egocentric human and vision-language data. Additionally, we endow it with a memory that gives the policy access to its recent past at negligible cost: a few register tokens in the action expert summarize the output of the VLM at each frame and persist in the expert's cache. This memory improves success in simulation and on real robots, and makes oscillations between poses about four times rarer. REKI uses a simple model design and training paradigm, and is optimized to be efficient at training and inference. We release the full code across data curation, training, inference and evaluation, deployment, and all pretrained and finetuned model weights.
REKI (ours)Qwen-RobotManipABot-M0 / M0.5π0.5RLDX-1GR00T-N1.5 / 1.6 / 1.7Other VLAs(InternVLA-A1.5, Being-H0.7, ACoT-VLA)
Success rate against model size. REKI sets a new Pareto frontier between performance and model size: it scores highest on LIBERO-Plus, and is only outperformed by models about twice its size on RoboTwin Clean2Rand and RoboCasa365. Hover over a point or a legend entry to identify the models.
Autonomous rollouts of REKI on a UR5 arm, at 3× speed. REKI is finetuned for 3k steps (1h of training) on 50 demonstrations per task.
Intuitively, past context should help a robot act, but it is not clear how best to use it in a VLA. Current methods put past frames into the vision-language backbone, unroll the model during training, or add a separate memory module, and each of these adds training or inference cost. REKI puts the memory in the action expert: a small set of register tokens summarizes each encoded frame and is cached across chunks, so each frame is encoded only once, with no separate module and no sequential unroll.
How VLAs carry the past, over one training window of three frames: (a) past frames in the VLM, (b) a recurrent state, (c) an added memory module. (d) In REKI, the memory tokens are registers in the action expert, read by the self-attention already present. The window is one batched pass, and every frame receives a loss.
A Qwen3.5-2B vision-language backbone encodes the camera views and the instruction as a context, which every block of a 200M-parameter flow-matching action expert reads by cross-attention. Robot state and flow time modulate the actions through adaptive normalization. In the expert, the action tokens of each frame are preceded by 32 learned register tokens. The registers cross-attend the encoded prompt and views of that frame, and self-attend the registers of the earlier frames. Since they do not attend the action tokens, they depend only on the observations and can be shared across frames and denoising steps.
Left: REKI. The register keys and values of past frames are kept in a cache and read by self-attention. Right: one expert block. Action tokens read the current and past registers and themselves, and registers never read them.
We train on windows of K = 3 frames from one episode, taken one second apart. At inference, the keys and values of the registers are cached in a rolling window over the K − 1 preceding frames: when a new frame enters, the registers of the oldest one are dropped.
1One step2Memory over time0:00 / 1:03
One step, then memory over time. The registers summarize the current frame and stay fixed during denoising. Their keys and values are cached, and the actions of later frames read them.
The memory is cheap at training and at inference.
Time of one action chunk on one H100 against the memory window K (two 256×256 cameras, ten Euler steps, VLM and action expert as CUDA graphs; median of 400 chunks, whiskers 10th to 90th percentile). Labels give the change against K = 1. Hover over a bar for its split.
We time one call, from the raw observation to the actions of one chunk, on one H100 with LIBERO inputs, for each model with its released checkpoint and code. Most of these models are released without a compiled path, or with one that can be further optimized, so we also compile them ourselves without changing their computation. With compilation, REKI predicts a chunk in 21.8 ms, the fastest of the compared models: X-VLA takes 25.5 ms, π0.5 (openpi) 33.8 ms, ABot-M0 46.1 ms and InternVLA-A1.5 51.3 ms. InternVLA-A1.5 uses the same VLM as REKI, but its larger action expert increases the denoising time.
1.4 s
GPU timehost timecompiled by us
Inference latency of one action chunk on one H100, from the raw observation to the actions, with LIBERO inputs. Dark: GPU time; light: host time. Dashed: compiled by us.
REKI is pretrained once, then finetuned on each benchmark separately:
Among the models pretrained on public data, REKI has the highest out-of-distribution success on LIBERO-Plus and RoboTwin Clean2Rand, although it is the smallest. On RoboCasa365, it reaches 38.4, ahead of the larger RLDX-1 (36.0) and Qwen-RobotManip (35.9), both pretrained on in-house data.
020406080success rate (%)With pretrainingNo pretrainingQwen-RobotManip (4.7B)68.5REKI (ours) (2.4B)65.8π0.5 (3.3B)63.8ABot-M0 (4.6B)55.7GR00T-N1.7 (3.1B)35.5Qwen-RobotManip (4.7B)52.3REKI (ours) (2.4B)52.0StarVLA (4.5B)36.6
LIBERO-PlusOverallCameraRobot initLanguageLightBackgroundNoiseLayout
406080100success rate (%)With pretrainingNo pretrainingREKI (ours) (2.4B)90.4Qwen-RobotManip (4.7B)89.0RLDX-1 (6.9B)86.7ACoT-VLA (3.8B)86.6π0.5 (3.3B)85.7InternVLA-A1.5 (2.7B)84.8Being-H0.7 (3B)84.0ABot-M0.5 (5B)83.7ABot-M0 (4.6B)80.5π0 (3.2B)69.4REKI (ours) (2.4B)86.1Qwen-RobotManip (4.7B)78.3StarVLA (4.5B)75.0
RoboCasa365AverageAtomicComposite seenComposite unseen
0306090success rate (%)Xiaomi-Robotics-1 (5.1B)57.4ABot-M0.5 (5B)40.4REKI (ours) (2.4B)38.4RLDX-1 (6.9B)36.0Qwen-RobotManip (4.7B)35.9GR00T-N1.5 (2.4B)23.9GR00T-N1.6 (3.0B)21.9π0.5 (3.3B)16.9
public datapretrained on in-house dataworld-action model
Success rate on RoboTwin Clean2Rand, LIBERO-Plus and RoboCasa365. Bars give the mean, ticks the conditions (RoboCasa365: the three task groups). Pick a condition to rank the models on it.
Pretraining on public data helps most out of distribution, and most where REKI without pretraining fails: it adds 13.8 points on Clean2Rand, including 30.3 under clutter and 22.4 under the Hard condition, and 4.3 on LIBERO-Plus.
Memory improves success on all three benchmarks, and most of the gain already comes with one past frame. Removing the past frames at evaluation brings the model back to the level of a memoryless one, so the gain comes from reading the past at inference.
Memory window, written as frames at finetuning → frames at evaluation: success rate. RoboCasa365: the bar gives the three task groups, the line the average. Hover over a bar for its values.
Without memory, the policy often gets stuck moving an arm back and forth between two poses, switching direction at each new chunk.
Oscillation without memory. Two RoboTwin rollouts of REKI trained without memory. Frames are one chunk (one second) apart, and A and B mark the two poses the arm alternates between.
With memory, such oscillations are about four times rarer on RoboTwin (4.1% of the episodes against 18.6%), and the difference is largest in two-arm tasks.
Oscillation. Left: share of oscillating episodes on RoboTwin Clean, by memory window. Right: oscillating episodes by task class, without (1 → 1) and with (3 → 3) memory.
We attribute this to perceptual aliasing: if two poses each look like demonstrations that continue towards the other, a memoryless policy alternates between them. A short history resolves the ambiguity.
REKI is pretrained for 200k steps on 32 H100 GPUs, on a mixture of public data only:
We keep each dataset's native control convention rather than remapping to a shared action space, and we curate every source before training. As REKI is trained only on public data, it is fully reproducible by the scientific community.
ModelStageRobotBenchmarkkyutai/reki-2bPretrainedpretraining mixture–kyutai/reki-2b-liberoFinetunedFranka PandaLIBERO, LIBERO-Pluskyutai/reki-2b-robotwin-cleanFinetunedALOHA (bimanual)RoboTwin Clean2Randkyutai/reki-2b-robotwin-cleanrandFinetunedALOHA (bimanual)RoboTwin 2.0, clean + randomizedkyutai/reki-2b-robocasa365FinetunedPandaOmron (mobile)RoboCasa365kyutai/reki-2b-robocasa-gr1FinetunedFourier GR-1 humanoidRoboCasa-GR1 tabletop
The code covers training, evaluation on all benchmarks, the data processing pipeline and a real-robot policy server.
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。