NVIDIA opens VFlash robot-planning code; model weights to follow

NVIDIA has released training, inference and evaluation code for VFlash, its visual-planning research for robots and autonomous vehicles. A November 7 release commit in the official Eagle repository lets researchers modify coordinate generation and compare it with conventional decoding. Trained target and drafter checkpoints are still forthcoming. The model license limits use to noncommercial research or evaluation.

The system predicts object locations and movement paths from camera observations and a task instruction. Training first fine-tunes a target model on spatial answers, then freezes it while teaching a five-layer drafter. The smaller model proposes a complete path in parallel, reducing the sequential steps used to spell out coordinates. The drafter is paired with the specific target that supplies its visual and language context.

The published data format pairs observed images or video with a question and an annotated spatial answer. Both training stages use that format. Robot trajectories use a camera coordinate frame; driving trajectories use vehicle-relative positions and headings. Prompts specify units and ranges, and dual-arm examples encode separate left and right paths. Coordinates use number tokens from 0 to 1,000, and inputs can identify a specific video frame.

Acceptance rules depend on the output. A trajectory that satisfies its coordinate schema can be committed as a whole, while ordinary text is checked against the target model. Bounding boxes receive an additional confidence check, with uncertain coordinates regenerated sequentially. Spatial accuracy is evaluated separately from structural validity.

What the timing result includes

In the paper submitted November 5, the authors report reducing per-step MolmoSpaces model latency on Jetson Thor from 3,552.7 milliseconds to 703.3 milliseconds at batch size one. The total covers vision, prefill, decoding and the action model. It excludes model loading and CPU preprocessing outside the instrumented modules; physical robot task duration was not measured.

The release’s results document specifically excludes the downstream action models and the runtimes used for the paper’s latency measurements. Its separate dense-motion target backend is also outside this release. The available training and evaluation code therefore covers a narrower set of components than the complete robot and driving experiments described in the paper.

The evaluation guide covers robot trajectories, driving, dense point motion and bounding boxes. Benchmark datasets must be obtained separately. Robot predictions need the expected arm identities and waypoint counts, without interpolation or repair. Valid outputs are then scored for measures including average distance from the reference path. Average and final-waypoint errors describe differences along the path and at its endpoint, in meters.

Dense-motion quality comparisons use another condition: the released runner selects the lowest-error result from five drafter candidates using visible ground truth. The conventional target produces one greedy answer. That reference-based candidate selection belongs alongside the quality scores, separately from timing a single generated candidate.

Target–drafter matching is explicit. Saved drafter configurations identify the target weights, tokenizer and text configuration used during training, and loading checks their compatibility. The reference training launch spans eight nodes with eight GPUs each. Drafter training also requires an environment with compiled FlexAttention support.

The code uses Apache 2.0, while models carry a separate NVIDIA license. Code redistribution requires a license copy and prominent notices on modified files. Models follow separate conditions for use, modification and distribution, including retaining the full license and existing notices on redistribution. Examples take locally prepared target and drafter checkpoint paths; their public release schedule remains unspecified.