Runway releases Praxis-1 robot control model with open weights

Video AI company Runway released the weights and code for Praxis-1, its robot control model, on November 11. The company introduced the model to a group of partners in September. Outside labs and companies can now download it, run their own evaluations and fine-tune it. Runway released the weights and accompanying code under Apache 2.0, which permits commercial use.

Praxis-1 predicts the movement a robot should make along with how the scene may look after it moves. Hugging Face hosts versions adapted to a DROID single-arm setup and a rig with two YAM arms, plus a base model for teams adapting it to different equipment. Runway built the system on the video pretraining it uses for its generative models. A separate component reads features from that video model to calculate the robot’s motion.

555 successes in 1,200 simulated trials

NVIDIA’s RoboLab-120 tests a single arm on 120 tabletop tasks in simulation. Runway evaluated the model it fine-tuned for DROID on ten trials per task. It completed 555 of 1,200 trials, or 46.2% as shown on NVIDIA’s leaderboard. Black Forest Labs’ FLUX 3 Action completed 515 on the same trial count, for 42.9%. On tasks NVIDIA classifies as simple, Praxis-1 succeeded on 300 of 640 trials. It completed 68 of 170 trials in the complex category.

Runway ran a separate comparison on physical hardware. On November 2, the team tested ten single-object tasks at a Franka FR3 station operated by Positronic. It ran each task ten times on the same rig and reported 88 successes for Praxis-1 and 80 for FLUX 3 Action. Runway used the ten single-object tasks published by Black Forest Labs for FLUX 3 Action. The comparison on a physical arm is separate from NVIDIA’s 120-task simulation.

Different arms and cameras call for adaptation

Runway’s single-arm checkpoint targets a DROID setup with a camera watching the workspace and another on the robot’s wrist. The dual-arm checkpoint is designed for two YAM arms. Runway’s repository describes using the base checkpoint for new equipment and training it on observations and actions for that robot. A change in the arm’s range of motion or how its gripper closes changes the commands the model has to produce.

The technical report describes sending the model an instruction, recent camera frames and the robot’s pose. It returns about 1.6 seconds of motion at a time. After the arm moves, the next call starts with new camera frames and an updated pose. That means the model sees what happened when a grasp missed. Runway reports that reaching for the wrong object was a common failure when two objects looked alike. In those cases the model often pictured the wrong target in its predicted video as well.

Runway also adapted Praxis-1 to Google’s RT-1 mobile manipulator using recordings from that robot after leaving them out of its earlier training mix. In a separate four-task simulation called SimplerEnv, Runway reported an average of 80.0% for the adapted model and cited a published result of 52.4% for RT-1. The experiment illustrates a transfer that used training data from the new hardware.

The default example calls for 80 GB of GPU memory

Runway’s GitHub quick start runs the single-arm checkpoint in simulation and calls for one CUDA GPU with at least 80 GB of memory. The repository describes a lower-precision option on selected workstation hardware. The repository says that option has yet to be tested on those GPUs in a loop that uses fresh observations after each movement.

Hugging Face documents the three checkpoints in its LeRobot library as well. The published training code lets teams update the full model or train a smaller set of added weights. Runway’s example for adapting the full base model to a new rig uses eight GPUs. Its example for adapting the DROID model to a new task on the same rig uses one 80 GB GPU and updates a smaller set of weights.