Official code release for ICCV 2021 paper SNARF: Differentiable Forward Skinning for Animating Non-rigid Neural Implicit Shapes. We propose a novel forward skinning module to animate neural implicit shapes with good generalization to unseen poses.
Update: we have released an improved version, FastSNARF, which is 150x faster than SNARF. Check it out here.
If you find our code or paper useful, please cite as
@inproceedings{chen2021snarf,
title={SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes},
author={Chen, Xu and Zheng, Yufeng and Black, Michael J and Hilliges, Otmar and Geiger, Andreas},
booktitle={International Conference on Computer Vision (ICCV)},
year={2021}
}
Clone this repo:
git clone https://github.com/xuchen-ethz/snarf.git
cd snarf
Install environment:
conda env create -f environment.yml
conda activate snarf
python setup.py install
Download SMPL models (1.0.0 for Python 2.7 (10 shape PCs)) and move them to the corresponding places:
mkdir lib/smpl/smpl_model/
mv /path/to/smpl/models/basicModel_f_lbs_10_207_0_v1.0.0.pkl lib/smpl/smpl_model/SMPL_FEMALE.pkl
mv /path/to/smpl/models/basicmodel_m_lbs_10_207_0_v1.0.0.pkl lib/smpl/smpl_model/SMPL_MALE.pkl
Download our pretrained models and test motion sequences:
sh ./download_data.sh
Run a quick demo for clothed human:
python demo.py expname=cape subject=3375 demo.motion_path=data/aist_demo/seqs +experiments=cape
You can the find the video in outputs/cape/3375/demo.mp4 and images in outputs/cape/3375/images/. To save the meshes, add demo.save_mesh=true to the command.
You can also try other subjects (see outputs/data/cape for available options) by setting subject=xx, and other motion sequences from AMASS by setting demo.motion_path=/path/to/amass_modetion.npz.
Some motion sequences have high fps and one might want to skip some frames. To do this, add demo.every_n_frames=x to consider every x frame in the motion sequence. (e.g. demo.every_n_frames=10 for PosePrior sequences)
By default, we use demo.fast_mode=true for fast mesh extraction. In this mode, we first extract mesh in canonical space, and then forward skin the mesh to posed space. This bypasses the root finding during inference, thus is faster. However, it's not really deforming a continuous field. To first deform the continuous field and then extract mesh in deformed space, use demo.fast_mode=false instead.
Install kaolin for fast occupancy query from meshes.
git clone https://github.com/NVIDIAGameWorks/kaolin
cd kaolin
git checkout v0.9.0
python setup.py develop
Download the AMASS dataset. We use ''DFaust Snythetic'' and ''PosePrior'' subsets and SMPL-H format. Unzip the dataset into data folder.
tar -xf DFaust67.tar.bz2 -C data
tar -xf MPILimits.tar.bz2 -C data
Preprocess dataset:
python preprocess/sample_points.py --output_folder data/DFaust_processed
python preprocess/sample_points.py --output_folder data/MPI_processed --skip 10 --poseprior
Run the following command to train for a specified subject:
python train.py subject=50002
Training logs are available on wandb (registration needed, free of charge). It should take ~12h on a single 2080Ti.
Run the following command to evaluate the method for a specified subject on within distribution data (DFaust test split):
python test.py subject=50002
and outside destribution (PosePrior):
python test.py subject=50002 datamodule=jointlim
You can use the trained model to generate animation (same as in Quick Start):
python demo.py expname='dfaust' subject=50002 demo.motion_path='data/aist_demo/seqs'
Download the CAPE dataset and unzip into data folder.
Run the following command to train for a specified subject and clothing type:
python train.py datamodule=cape subject=3375 datamodule.clothing='blazerlong' +experiments=cape
Training logs are available on wandb. It should take ~24h on a single 2080Ti.
You can use the trained model to generate animation (same as in Quick Start):
python demo.py expname=cape subject=3375 demo.motion_path=data/aist_demo/seqs +experiments=cape
We use the pre-processing code in PTF and LEAP with some adaptions (./preprocess). The network and sampling part of the code (lib/model/network.py and lib/model/sample.py) is implemented based on IGR and IDR. The code for extracting mesh (lib/utils/meshing.py) is adapted from NASA. Our implementation of Broyden's method (lib/model/broyden.py) is based on DEQ. We sincerely thank these authors for their awesome work.