Sim-to-Thor

1Motivation

Too slow at the edge

GR00T N1.7 runs at about 8 Hz on Thor in BF16 (125 ms), while reactive manipulation needs 30 Hz or more.

Costly real data

Robot demonstrations are expensive, and plain simulation frames look visibly different from camera images.

Hard-to-trust evaluation

Real-robot results are slow to collect and hard to reproduce.

2Key results expected

8.3×lower end-to-end latency on Thor (125 → 15 ms)

67 Hzpolicy rate with async execution and RTC/LIPO

< 2 %pLIBERO success drop after full compression

3NVIDIA stack

GR00T N1.7base VLA policy

Cosmos 3photoreal sim-to-real rendering, world-action policy

Isaac Simscene generation, latency-aware evaluation

Jetson AGX Thoron-robot inference

TensorRTNVFP4 engines, CUDA Graph

4Pipeline overview

Sim-to-real data synthesisIsaac Sim scenes → Cosmos 3 photoreal rendering
Policy training & compressionfine-tune → step distillation → vision token pruning → NVFP4
Real-time control on ThorTensorRT + CUDA Graph, async chunks with RTC/LIPO

4.1 Data synthesis

Isaac Sim recreates LIBERO-style and SO-101 move-the-cube scenes with randomized objects, lighting and layout.
Cosmos 3 turns sim frames into camera-like images to shrink the visual sim-to-real gap.
Evaluation scenes are held out from all training scenes.

4.2 Policy compression

Step distillation

A flow-matching action head integrates the velocity field over several denoising steps. We distill it into a single step with a Drift loss: each generated action is pulled toward the demonstration while sibling samples of the same input push each other apart, which keeps the action distribution diverse without a teacher at inference time.

GR00T N1.7: the DiT action head runs once instead of 4 times.
Cosmos3-Edge: the 4-step sampler with classifier-free guidance (8 tower forwards) becomes 1 forward at σ = 1 with guidance off.

Vision token pruning / video token drop

Most of the sequence entering the backbone is visual. We remove the tokens the action does not need.

GR00T N1.7 · Grid-Sampler: a small head predicts K continuous coordinates per camera and bilinearly samples K tokens from the vision-encoder patch grid, so only K image tokens reach the LLM.
Cosmos3-Edge · video token drop: the world-action model also generates future video (8 latent frames × 340 tokens). We keep the conditioning frame and the first K generated frames and drop the rest; dropping 75% keeps 2 of 8 frames, dropping 100% keeps actions only (FastWAM).

NVFP4 quantization

Mixed NVFP4/FP8 TensorRT engines, calibrated per component: vision encoder in FP8, LLM in NVFP4 with o_proj/down_proj in FP8, DiT action head in NVFP4.

4.3 Real-time control on Thor

Per-component TensorRT engines captured in CUDA Graphs remove launch overhead.
Async execution: the robot runs chunk k while Thor computes chunk k+1; RTC/LIPO keeps transitions smooth.

Sync

inferexecuteinfer

Async

execute kexecute k+1

infer k+1infer k+2

5Evaluation protocol

Latency-aware simulation: Sim time keeps running during inference (Thor in the loop, or measured latency injected).
Benchmarks: GR00T N1.7: LIBERO (4 suites, 500 episodes each) and SO-101 move-the-cube on the real robot (20 trials per config). Cosmos3-Edge: RoboLab (96 episodes: 12 tasks × 8).
Trust in sim: Sim vs. real success correlation (Pearson r, MMRV) across all configs.

6Results

6.1 Latency on Thor expected

BF16 TRT125

NVFP4 (NVIDIA)39.9

+ step distill28

+ token prune19

Ours (+Graph, RTC)15

ms per decision. 125 ms and 39.9 ms come from NVIDIA's Jetson AI Lab GR00T N1.7 on Thor tutorial; the other rows are targets.

6.2 Success rate by vision token pruning (VTP) and step distillation (SD)

Summary · success rate (%) as shown on the poster

Config	GR00T N1.7		Cosmos3-Edge
Config	LIBERO avg	SO-101	RoboLab overall
Baseline	TBD	TBD	36.5
+ VTP	TBD	TBD	38.5
+ SD	TBD	TBD	22.7
+ VTP + SD	TBD	TBD	21.9

VTP: Grid-Sampler for GR00T, 75% generated-video token drop for Cosmos3-Edge. SD: one-step Drift distillation. TBD = being measured.

Detailed results

GR00T N1.7 · LIBERO success rate (%), 500 episodes per suite

Config	Spatial	Object	Goal	Long	Avg
Baseline (4-step flow matching)	TBD	TBD	TBD	TBD	TBD
+ VTP (Grid-Sampler, K tokens/camera)	TBD	TBD	TBD	TBD	TBD
+ SD (Drift, 1 step)	TBD	TBD	TBD	TBD	TBD
+ VTP + SD	TBD	TBD	TBD	TBD	TBD

GR00T N1.7 · SO-101 move-the-cube on the real robot

Config	Success / trials	Success (%)
Baseline	TBD / 20	TBD
+ VTP	TBD / 20	TBD
+ SD	TBD / 20	TBD
+ VTP + SD	TBD / 20	TBD

Cosmos3-Edge · RoboLab success rate (%), 96 episodes (12 tasks × 8)

Config	Overall	Simple	Moderate	Complex
Baseline (4-step, CFG)	36.5	56.2	34.4	18.8
+ VTP: video token drop 75%	38.5	65.6	25.0	25.0
video token drop 100% (FastWAM)	25.0	46.9	28.1	0.0
+ SD: 1-step Drift	22.7	43.6	24.6	0.0
+ VTP + SD: 1-step Drift + drop 75%	21.9	40.6	25.0	0.0
1-step Drift + drop 100%	9.4	3.1	25.0	0.0

Dropping 75% of the generated video tokens matches the baseline. One-step distillation loses success on complex tasks.

6.3 Synthetic data expected

Real only52

+ Isaac Sim61

+ Isaac Sim + Cosmos 370

Sim-real correlation: r = 0.62 (Isaac Sim) → 0.91 (+ Cosmos 3).

★Rollout videos

LIBERO-Spatial

LIBERO-Object

LIBERO-Goal

LIBERO-Long

7Limitations & next steps

Real-robot results cover one task (move-the-cube) on SO-101.
One-step distillation still loses success on complex RoboLab tasks; combining it with video token drop needs further training.
Next: Cosmos 3 planning on B300 with the Thor policy for long-horizon tasks.