@inproceedings{mocanet2022,
title={MoCaNet: Motion Retargeting in-the-wild via Canonicalization Networks},
author={Zhu, Wentao and Yang, Zhuoqian and Di, Ziang and Wu, Wayne and Wang, Yizhou and Loy, Chen Change},
booktitle={AAAI},
year={2022}
}
MoCaNet: Motion Retargeting in-the-wild via Canonicalization Networks
Peking University
Shanghai AI Lab
SenseTime Research
Southeast University
Nanyang Technological University
Abstract
We present a novel framework that brings the 3D motion retargeting task from controlled environments to in-the-wild scenarios. In particular, our method is capable of retargeting body motion from a character in a 2D monocular video to a 3D character without using any motion capture system or 3D reconstruction procedure. It is designed to leverage massive online videos for unsupervised training, requiring neither 3D annotations nor motion-body pairing information. The proposed method is built upon two novel canonicalization operations, structure canonicalization and view canonicalization. Trained with the canonicalization operations and the derived regularizations, our method learns to factorize a skeleton sequence into three independent semantic subspaces, i.e., motion, structure, and view angle. The disentangled representation enables motion retargeting from 2D to 3D with high precision. Our method achieves superior performance on motion transfer benchmarks with large body variations and challenging actions. Notably, the canonicalized skeleton sequence could serve as a disentangled and interpretable representation of human motion that benefits action analysis and motion retrieval.
Canonicalization
The first row is the input 2D skeleton sequences. The following rows show the 3D results after applying view canonicalization, structure canonicalization, and both. Structure canonicalization yields skeleton sequences with a uniform body structure, while the motion and other features are preserved. Similarly, view canonicalization provides skeleton sequences with the same view angle, casting different sequences to a uniform view.
Citation
W. Zhu*, Z. Yang*, Z. Di, W. Wu+, Y. Wang, C. C. Loy. "MoCaNet: Motion Retargeting in-the-wild via Canonicalization Networks." Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI), 2022.
Bibtex: