Joints, Camera, Action Co-Generating 3D Motion and Video for Arbitrary Rigs

Joints, Camera, Action! Co-Generating 3D Motion and Video for Arbitrary Rigs

Noam Rotstein, Yoad Tewel, Chen Tessler, Haggai Maron, Gal Chechik, Or Litany, Yoni Kasten, Yuval Atzmon

TL;DR: Joints, Camera, Action! (JCA) jointly creates video and synchronized 3D motion for rigs of any topology from a text prompt. An input image enables motion that responds to its surroundings and can be registered within the 3D scene. Adding a few chosen keyframes provides control over the motion trajectory. JCA also turns rig motion into video and video into rig motion.

Video generation models can now depict remarkably realistic motion. Yet it is unclear how to use this motion beyond pixels: to animate a specific 3D asset and replay the animation in its 3D scene. For instance, a video model can show a monkey jumping over a crossbar, but it cannot return that jump as replayable 3D motion placed in the 3D scene. Here we present a model that directly co-generates the video and 3D motion for any supplied rig, along with a camera that registers this motion to the scene. Prior video–motion models either handle only human skeletons, often adding billions of parameters, or triangulate 3D motion from generated 2D pose maps. We instead keep 3D motion in a separate model that supports arbitrary skeletons, and connect it to a pretrained video model through a small set of cross-attention layers. This separation lets the motion model first learn how skeletons move from motion-only data, which is easier to obtain and augment than paired video. Additionally, our model learns to locate each joint in the video frames, which lets us recover the camera and place the 3D motion in the scene. To train across many skeletons, we use a generative video editor to place animated 3D assets in scenes suited to their actions, curating the edits so their known motion remains valid. Together with human and robot video, this yields nearly 200K training pairs. Our evaluations demonstrate that our model animates diverse rigs from text more faithfully than video-first pipelines, and its motion responds to the surrounding scene. It also turns video into rig motion, rig motion into video, and enables keyframe-based control of the 3D motion.

Given a rig in its rest pose, an input image, and a text prompt, JCA generates video and 3D rig motion that responds to the scene.

Given a rig in its rest pose, a text prompt, and RGB keyframes, JCA generates the video and 3D rig motion between them. Playback briefly stops at each input keyframe for comparison.

Given a video and the subject’s rig in its rest pose, JCA recovers the subject’s 3D motion.

Generate a video from a rig in its rest pose, a 3D motion sequence, and a text prompt, without conditioning on a first frame.

Generate a video from a rig in its rest pose, a 3D motion sequence, and a text prompt, using a reference image to specify the first frame.

Examples of the video-and-motion and motion-only training data.

Text-driven rig animation with JCA, AnimateAnyMesh, Puppeteer, and ActionMesh. Animations are generated from the same assets and action prompts.

@misc{rotstein2026jointscameraactioncogenerating, title={Joints, Camera, Action! Co-Generating 3D Motion and Video for Arbitrary Rigs}, author={Noam Rotstein and Yoad Tewel and Chen Tessler and Haggai Maron and Gal Chechik and Or Litany and Yoni Kasten and Yuval Atzmon}, year={2026}, eprint={2610.28632}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2610.28632}, }

Dark mode

Enable JavaScript to view these examples, or open the original supplementary page.