Learning-based approaches to monocular motion cap- ture have recently shown promising results by learning to regress in a data-driven manner. However, due to the chal- lenges in data collection and network designs, it remains challenging for existing solutions to achieve real-time full- body capture while being accurate in world space. In this work, we introduce ProxyCap, a human-centric proxy-to- motion learning scheme to learn world-space motions from a proxy dataset of 2D skeleton sequences and 3D rotational motions. Such proxy data enables us to build a learning- based network with accurate world-space supervision while also mitigating the generalization issues. For more accu- rate and physically plausible predictions in world space, our network is designed to learn human motions from a human-centric perspective, which enables the understand- ing of the same motion captured with different camera tra- jectories. Moreover, a contact-aware neural motion descent module is proposed in our network so that it can be aware of foot-ground contact and motion misalignment with the proxy observations. With the proposed learning-based solu- tion, we demonstrate the first real-time monocular full-body capture system with plausible foot-ground contact in world space even using hand-held moving cameras.

Fig 2. Illustration of the proposed method ProxyCap. Our method takes the estimated 2D skeletons from a sliding window as inputs and estimates the relative 3D motions in the human coordinate space. These local movements are accumulated frame by frame to recover the global 3D motions. For more accurate and physically plausible results, a contact-aware neural motion descent module is proposed to refine the initial motion predictions.