GNNRetarget: Vision-Guided Motion Retargeting Based on Graph Neural Network for Dexterous Robot

Motion retargeting is pivotal in executing complex tasks by transferring human motion to robotic systems with precision. However, existing motion retargeting approaches often require elaborate human motion capture setups and fail to account for the scale discrepancies between hands and bodies, resulting in suboptimal accuracy. Additionally, their retargeting methods face challenges such as high computational complexity, susceptibility to local minima, and inaccuracies in mapping unseen motions. Moreover, these methods are typically tailored to specific robot platforms, lacking generalization capabilities. In this paper, we propose a novel vision-guided motion retargeting framework and approach to address these limitations. Our framework employs a dual-stream architecture, processing hands and bodies separately to effectively mitigate scale discrepancies, thereby enhancing precision. The proposed retargeting method integrates a graph encoder network to generate meaningful initial embeddings, subsequently optimized in the latent space. This strategy significantly reduces complexity while circumventing local minima issues. Crucially, by graphically modeling human poses and robot files, our method eliminates the need for paired datasets, enabling broad applicability across diverse robots. Experiments successfully replicate human motions and validate the feasibility and accuracy of human-robot motion retargeting in both simulated environments on RMC-DA, YuMi, and Unitree H1, as well as the real-world RMC-DA, underscoring the practical value of our vision-guided motion retargeting framework.