NEXUS: Perceptive Whole-Body Control for Terrain-Adaptive Teleoperation

Abstract

NEXUS enables whole-body humanoid teleoperation when the human operator and robot stand on different terrains. A scalable motion adaptation algorithm adjusts contacts and posture to generate paired flat-ground and terrain-adapted references without tuning individual motions or terrains. Using nearly 1,000 hours of paired motions, a teacher-student framework trains a perceptive controller that combines human motion commands with onboard sensory feedback. The resulting policy preserves diverse commanded behaviors while adapting to local terrain, improving tracking and terrain generalization over existing whole-body controllers. Zero-shot deployment demonstrates real-time teleoperation across previously unseen real-world terrains.