ObjRetarget: An Object-Aware Motion Retargeting Framework with Anthropomorphic Arm Constraints and Polyhedral Hand Modeling

Learning robot dexterous manipulation from human manipulation videos requires reliably retargeting human intent to executable robot actions while maintaining stable hand–object contact, which remains a key challenge in embodied intelligence. Existing retargeting methods often ignore explicit contact modeling or rely on reinforcement learning, resulting in limited accuracy and generalization. To address this, we propose ObjRetarget, a human-to-robot motion retargeting framework for learning robot dexterous manipulation from human videos, which integrates anthropomorphic arm trajectory constraints with structured hand–object geometric modeling. For arm motion, reference trajectories extracted from human videos are used for initialization, followed by anthropomorphic constraints and redundancy-aware optimization to generate natural and accurate movements. For hand manipulation, ObjRetarget represents multi-finger contacts using polytope clusters and preserves contact structure through geometric invariants to improve stability. Experiments on real robots show that ObjRetarget improves manipulation success rates and contact stability across multiple dexterous tasks, and generalizes well to different demonstrations, object poses, and task settings.

For each task, the top row shows the human demonstration and the bottom row shows the corresponding robot execution. From left to right: (a) Sequential Place: sequentially placing two cartons onto a tray, (b) Soft Place: grasping a plush toy and placing it into a basket, (c) Drawer Close: placing a medicine box into an open drawer and closing it, (d) Fruit Place: transferring a mango from the table to a fruit basket, (e) Pour Water: picking up a bottle, pouring water into a container, and returning it, and (f) Bimanual Place: bimanual grasping of an apple and a lemon followed by synchronized placement.