AnyDexRT: Calibration-Free Dexterous Hand Retargeting with Few-Shot Human Guidance

Teleoperation is a key interface for controlling dexterous robotic hands and collecting demonstrations for imitation learning. Its effectiveness largely depends on kinematic retargeting, which maps operator hand motions to feasible and intuitive robot hand motions. Existing methods often require hand-crafted objectives, precise calibration, or global shape matching between human and robot hand spaces, making them sensitive to hand-specific tuning and less reliable across different dexterous hands. We propose AnyDexRT, a calibration-free retargeting method for intuitive dexterous teleoperation across human-like dexterous hands. AnyDexRT combines self-supervised fingertip correspondence learning with few-shot human guidance to anchor the mapping in task-relevant regions, and further refines pinch-related poses using a contact classifier. Experiments on diverse dexterous hands and real-world teleoperation tasks show that AnyDexRT improves retargeting quality, reduces manual tuning, and provides more intuitive and efficient control than prior retargeting methods.

AnyDexRT learns fingertip-level correspondence via self-supervised shape matching and uses few-shot human guidance to anchor the mapping in task-relevant regions, reducing ambiguity caused by redundant robot-hand motion spaces. It further refines pinch-related poses with a contact classifier to improve the performance of pinch motions. During deployment, the fingertip mapper produces retargeted positions, which are refined by the contact classifier and converted into robot joint commands.

We design several retargeting objectives to produce a geometrically consistent and intuitive mapping from the human fingertip space to the robot fingertip space.

Retargeting Objectives. (a) Step-by-step visualization of adding retargeting objectives. (b) Full Chamfer loss can force unnatural coverage of redundant robot fingertip regions. (c) Distance loss preserves the geometric structure of the retargeted space and reduces mapping distortion. (d) Local motion preservation encourages consistent motion directions between human and robot fingertips, and is less sensitive to calibration compared to global motion preservation. (e) Few-shot anchor alignment resolves mapping ambiguity and stabilizes retargeting.

To evaluate retargeting quality, we measure motion consistency, which reflects whether the robot hand responds consistently with the operator’s motion intent. We report global motion consistency (GMC) and local motion consistency (LMC) across 7 dexterous hands over 5 random seeds. GMC follows GeoRT and compares displacement directions in a shared coordinate frame, assuming an ideal, well-calibrated setting. LMC compares directions in local frames, making it less calibration-dependent and more aligned with operator control.

AnyDexRT achieves strong performance across different hand embodiments and random seeds, improving the average local motion consistency from 59.8% to 90.2% and maintaining competitive global motion consistency under an ideal, well-calibrated shared frame. This shows that AnyDexRT better preserves the operator’s motion intent, and provides more stable and general retargeting.