Meet ‘Code-as-World’: An Agentic Loop That Rewrites Real Videos Into Executable MuJoCo Physics Programs

What changed

Code-as-World can take a real-world video and convert it into editable, executable code for MuJoCo, a physics engine often used in robotics and AI research. It creates a verified digital replica of the scene captured in the video and then rewrites it as a MuJoCo program that faithfully simulates physical interactions from the original footage. This allows physical reasoning models to train on real-world scenarios expressed as manipulable code rather than raw pixels.

Why builders should care

This approach bridges the gap between real-world complexity and synthetic simulation environments by producing high-fidelity, editable simulations grounded in actual observations. Developers building AI agents or robotic systems need realistic training data that can be modified and verified. By converting video into executable physics scenes, Code-as-World gives builders a way to improve model training without relying solely on manually crafted or artificial data. It also automates scene reconstruction in ways that can speed up iterative testing and design.

The practical takeaway

Anyone working on embodied AI, robotics control, or physical scene understanding can use this to accelerate training cycles and validation. It pressures existing simulation workflows to incorporate real-world evidence more directly, reducing guesswork and hand-tuning. It also makes data collection more scalable—capturing video becomes a straightforward method to generate editable environments for training rather than just for evaluation or benchmarking. The agentic loop means simulation quality self-improves as the system learns from the real to enhance the virtual.

What to watch next

Watch for integrations of Code-as-World with existing robotics frameworks and AI pipelines that rely on MuJoCo or similar physics engines. Expanding this technique beyond MuJoCo to other simulation platforms could widen adoption and raise expectations about how realistic and testable physical environments must be. Also, keep an eye on whether this lowers barriers to building custom simulators by automatically mining real-world videos for complex scene setups instead of handcrafting them.

AI Quick Briefs Editorial Desk