hooneyskywalker/humanoid-motion-tracking-policies

Humanoid motion tracking policies

Whole-body control checkpoints that make a Unitree G1 humanoid follow human motion capture clips in simulation. Seventeen single-clip expert policies, plus one distilled policy that replaces fourteen of them.

Code: github.com/hooneyskywalker0127/humanoid-motion-tracking · Training curves: W&B · Videos: YouTube

Six seconds of walk2_subject4: the policy on the left, the reference it is following on the right, both in Isaac Lab.

Contents

Path	What it holds
student/	The distilled policy. final_model.pt, policy.onnx, config.json, evaluation JSONs
policies/<sequence>/	One expert per sequence, 17 of them. model_29999.pt, policy.onnx
eval/	Per-sequence expert evaluation, 100 rollouts each
eval_polysim/	The same rollouts scored with PolySim's five metrics
sym/	Every sequence re-scored in MuJoCo, with and without domain randomization
tables/	Rendered tables: per-sequence results, MuJoCo transfer, the two pass criteria
media/	GIFs and figures used on the GitHub page
igris_c/	Second robot IGRIS-C: training from scratch vs. transferring the G1 expert with Any2Any-style LoRA, on run2_subject4
reward_ablation/	Reward ablation on obstacles3_subject3: four 10,000-iteration policies, each with one pair of tracking rewards zeroed, and their evaluations

The distilled policy

Architecture — MLP, hidden [2048, 2048, 1024, 1024, 512], ELU. The capacity mattered: [1024, 512, 256] left one of the fourteen clips at 9% completion.

Input — 260 floats.

Slice	Width	Meaning
0:29	29	Reference joint positions
29:58	29	Reference joint velocities
58:67	9	Anchor (torso) pose error
67:160	93	Proprioception: base linear/angular velocity, joint positions, joint velocities, previous action
160:189	29	Reference − current joint position
189:218	29	Reference − current joint velocity
218:260	42	Reference − current body positions, in the anchor frame

Every velocity term is multiplied by vel_scale = 0.05 before it reaches the network. The last 100 entries are differences rather than raw references, following OpenTrack: handed the raw values, the network has to learn the subtraction first.

Output — 29 floats, a residual. The joint position target is

target = action + obs[0:29] - default_joint_pos

with default_joint_pos in student/config.json. The ONNX graph applies the residual internally, so policy.onnx takes obs [B, 260] and returns actions [B, 29] ready to use.

How to get started

import numpy as np, onnxruntime as ort

sess = ort.InferenceSession("student/policy.onnx")
obs = np.zeros((1, 260), dtype=np.float32)       # see the layout above
action = sess.run(["actions"], {"obs": obs})[0]  # (1, 29)

A complete loop that builds the observation from a MuJoCo state and steps the robot is src/sim2sim_student.py. The .pt checkpoints need BeyondMimic and Isaac Lab; the .onnx files need neither.

Training details

Experts — BeyondMimic, PPO, 4096 environments, 30,000 iterations, one policy per clip. Nineteen sequences passed the retargeting quality filter and seventeen were trained.

Student — HOVER's neural_wbc/student_policy, DAgger, fourteen teachers into one policy. The trainer, network, buffer and loss are unmodified; only the import paths and the single-teacher assumption were changed.

python scripts/distill/train_student.py \
  --num_envs 4096 --seqs configs/distill_seqs.txt \
  --max_iteration 50000 --save_iteration 10000 \
  --sampling_mode uniform --headless

50,000 iterations, 2 h 44 min on one RTX 5080. Isaac Sim 5.1, Isaac Lab 2.3.2, rsl-rl 3.1.2. Hyperparameters are in student/config.json.

Three of the seventeen experts are not teachers: their completion rate is 0%, so there is no successful trajectory to imitate. That is a reference problem, not a training one — those clips penetrate the floor or spend a quarter of their frames airborne. kobe is held out as a control.

Evaluation

Both rows below: 100 rollouts per clip, full clip length, domain randomization off, identical evaluation code. Each expert runs on its own clip; the student runs on all fourteen.

	Completion	E_g-mpbpe	E_mpbpe	E_mpjpe	E_mpbve	E_mpbae
14 experts, each on its own clip	99.00%	101.7 mm	41.7 mm	0.084	4.78	2.09
One distilled policy	99.71%	89.8 mm	40.6 mm	0.082	4.32	1.91

Metric definitions follow the GMR paper and, for the velocity and acceleration terms, PBHC. E_g-mpbpe is global body position error, E_mpbpe the same after aligning the root (drift removed, pose only), E_mpjpe joint angle error in radians, E_mpbve and E_mpbae body velocity and acceleration error in mm/frame and mm/frame². Error terms average over completed rollouts only, so read them next to the completion rate.

The distilled policy is lower on E_g-mpbpe for all fourteen clips. Eleven complete at 100%; the other three are walk1_subject1 and walk2_subject3 at 99% and jumps1_subject1 at 98%.

python scripts/distill/eval_student.py \
  --checkpoint student/final_model.pt --seqs configs/distill_seqs.txt \
  --num_envs 100 --student_hidden 2048 2048 1024 1024 512 --headless

Per-clip numbers for both sides are in student/eval_matched_100rollouts.json.

Also measured — MuJoCo transfer without retraining, 12 of 14 clips (sym/); 76.1% completion under domain randomization; motion-to-motion transitions without a reset.

Reward ablation

Which BeyondMimic tracking rewards hold the policy up? Same clip (obstacles3_subject3), same seed, 10,000 iterations each, one pair of rewards zeroed per run.

Condition	Completion	E_g-mpbpe	E_mpbpe
all rewards	63%	337 mm	60 mm
no global anchor pos/ori	3%	1255 mm	61 mm
no relative body pos/ori	0%	—	—
no global body lin/ang velocity	55%	956 mm	60 mm

Every pair is needed. Without the anchor terms the posture still matches (E_mpbpe unchanged) but the robot drifts off the reference; without the body-pose terms it falls within two seconds. The global velocity terms turn out to hold position too: during training the anchor error without them (2.1 m) is larger than without the anchor terms (0.95 m). 10,000 iterations is short for this clip — the all-rewards policy itself completes only 63% — so these are budget-matched, not converged, comparisons on one clip and one seed. Details in reward_ablation/README.md.

Second robot: IGRIS-C

IGRIS-C (1.5 m, 58 kg, 31 DoF) on one running clip, run2_subject4. A: PPO from scratch. B: the G1 expert for the same clip, frozen, adapted with LoRA following Any2Any (joint-slot alignment, LoRA on the actor and critic hidden layers, 5.0% trainable, the expert's PPO settings). Both 30,000 iterations.

Start frame	A completion	B completion
0	21%	0%
1000	96%	16%
4000	98%	35%
8000	98%	57%

B learns much faster early on (training episode length 172 at iteration 100, A 12.5) but plateaus lower. On this setup, training from scratch wins. Details and checkpoints in igris_c/. No IGRIS-C model files are included (the model has no license).

Per-clip policies for IGRIS-C and ROBOTIS AI Sapiens K1

Both robots are outside GMR's official list, so each needed its own IK config and reference versions. Layout: igris_c/<clip>/<ref>/model_29999.pt + eval/iter_29999.json, robotis_k1/<clip>/<ref>/.... The <ref> folder is the internal retargeting tag; the public names are IGRIS-C s90 = retargeting v4 (e31 = the failed candidate before it) and K1 p11 = retargeting v1 (p10 = the failed candidate). Same protocol as the G1 experts: PPO from scratch, 30,000 iterations, evaluated from frame 0, 100 rollouts, domain randomization off.

Clip	Robot	Reference	Completion	E_mpbpe	Note
dance2_subject3	IGRIS-C	e31	0%	—	reference demanded bigger jumps than the human (pelvis apex +0.40 m vs +0.33)
dance2_subject3	IGRIS-C	s90 (v4)	100%	48 mm	G1 100% / 45 mm
dance2_subject3	K1	p10	0%	—	same cause (+0.41 m)
dance2_subject3	K1	p11 (v1)	100%	41 mm
obstacles3_subject3	IGRIS-C	s90 (v4)	81%	44 mm	crawl is outside the 131° knee range (4.1% of frames); G1 98%
obstacles3_subject3	K1	p11 (v1)	94%	43 mm	knee range 135° (3.9% of frames)
run2_subject4	IGRIS-C	s90 (v4)	98%	60 mm	v3 72%; G1 99%
run2_subject4	K1	p11 (v1)	99%	67 mm	G1 99%
jumps1_subject1	IGRIS-C	s90 (v4)	97%	46 mm	v3 74%; G1 98%
jumps1_subject1	K1	p11 (v1)	98%	46 mm	G1 98%

v4/v1 recipe: GMR as in the paper (pelvis position weight 100, per-frame grounding, constant ground height 0.03, leg scale 0.90 for the heavier IGRIS-C) plus contact-gated foot weights 100/20, a trunk rotation weight (200 IGRIS-C, 100 K1) and an arm joint velocity limit (8 and 10 rad/s). The configs, the MJCF-to-config generator and the pre-training reference scorecard are in the GitHub repo (configs/gmr/, scripts/newrobot/, docs/new-robot.md). The remaining ten clips (walking, aiming) follow. No IGRIS-C or K1 model files are included; both come from their vendors' public repositories.

Limitations

• Simulation only. Everything here is Isaac Lab and MuJoCo. No G1 hardware was involved, and nothing here is a sim-to-real result.
• No generalization to unseen clips. On the remaining 63 LAFAN1 sequences the distilled policy completes none of them. It survives 13.4% of an unseen take of a seen motion type and 8.6% of an unseen motion type. Fourteen clips are far too few for coverage; this is a consolidation result, not a generalist one.
• Flat ground only. The reference motions assume a flat floor.
• The experts were trained on retargeted motion whose quality filter looked at foot error alone, which let three unusable references through.

Citation

@misc{humanoidmotiontracking2026,
  title  = {humanoid-motion-tracking: whole-body motion tracking policies},
  author = {Kim, Sehoon},
  year   = {2026},
  url    = {https://github.com/hooneyskywalker0127/humanoid-motion-tracking}
}

Built on BeyondMimic, HOVER and GMR. Motion data is LAFAN1 (Ubisoft La Forge). The license here covers the checkpoints; LAFAN1 and the upstream repositories carry their own.