MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpainting

NVIDIA Paper - "MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpainting"

📚 https://arxiv.org/pdf/2409.14393

Results 📊:

• Outperforms previous methods on full-body tracking (99.2% vs 97.1% success on test set)

• Achieves 98.1% success on VR tracking without task-specific training

• Demonstrates robust performance on irregular terrains (95.4% success)

• Enables new capabilities like text-to-motion synthesis and object interactions

Original Problem ⚠️:

Existing physics-based character animation approaches typically develop specialized controllers for specific tasks, lacking versatility and requiring complex reward engineering. A unified controller supporting diverse control modalities and generalizing across tasks remains an open challenge.

-----

Solution in this Paper 💡:

• Introduces MaskedMimic, a unified physics-based character controller framed as a motion inpainting problem

• Trains on randomly masked motion sequences to enable flexible conditioning on partial constraints

• Uses a conditional VAE architecture with a learned prior to model diverse plausible motions

• Employs a two-stage training process:

1) Fully-constrained motion tracking controller trained via RL

2) Partially-constrained controller distilled from the first stage using behavioral cloning

• Supports multi-modal inputs: keyframes, text commands, object interactions

-----

Key Insights from this Paper 💡:

• Formulating character control as motion inpainting enables flexible, intuitive user control

• A single unified model can handle diverse tasks without task-specific training

• Structured masking and episodic latent noise improve temporal coherence

• Residual prior architecture crucial for controlling the latent space