3DMM model fitting using Pytorch This is a fitting framework implemented in Pytorch for reconstructing the face in an image or a video using a 3DMM model. The framework only uses Pytorch modules and a differentiable renderer from pytorch3d. The whole module is differentiable and can be integrated into other systems for the gradient propagation. Updates :star2: Refactored the code to make future extension easier (other 3DMM models will be supported). The hyper-parameters are also re-organized. :star2: Added
support for monocular video reconstruction. :star2: Added support for multi-gpu when processing long videos. Installation Requirements pytorch3d It might require a specific version of Pytorch to make pytorch3d run succussfully on gpus, please follow the official instructions. Basel Face Model 2009 (BFM09) Expression Basis extra expression basis. Download the Basel Face Model and put "01_MorphableModel. Go to the repo, download the "CoarseData" and put "Exp_Pca. Convert the BFM parameters by: Single Image Reconstruction The results are stored
in "results" folder. Monocular Video Reconstruction To fit a video, simply run: The script will extract frames, detact features and fit all frames. Fitting a video is a bit different from fitting an image, because frames are not isolated. Then we estimate the other coefficients (expression, rotatation etc. ) for each frame and keep the shape and texture coefficients fixed. For the first frame, we use much more iterations to get a good starting point. For remaining frames, each is initialized
from the previous estimation. Pose regulerizers are also imposed to increase temporal consistency. Please check the reconstructed video in the result folder. Multi-gpu and -process support It could take quite a while to process a long video, so multi-gpu and multi-process are also supported to accelerate fitting. To use this feature, simply run: Here we use 4 gpus and 4 workers. We can also use more workers to assign each gpu with multiple workers. The video will
be evenly split and each clip will be fit respectively. Hyperparameters There are bunch of parameters that might require further tuning. Try to change it to find a trade-off between speed and accuracy. If the result is not good, try to play with the parameters. Acknowledgement The code is partially borrowed from Deep3DFaceReconstrution, which is a Tensorflow-based deep reconstruction method using CNNs. Please note that our framework does not require any pretrained deep models. We estimate the parameters
directly using the landmarks and photometric loss as the supervision.
常见问题
记录了多少星标和分叉？
ascust/3DMM-Fitting-Pytorch 有 653 个星标和 97 个分叉。
仓库排名是多少？
ascust/3DMM-Fitting-Pytorch 排名第 15,450。
记录的语言和类别是什么？
ascust/3DMM-Fitting-Pytorch 有 653 个星标，语言为 Python，类别为 Model Development。
This is a fitting framework implemented in Pytorch for reconstructing the face in an image or a video using a 3DMM model.
The framework only uses Pytorch modules and a differentiable renderer from pytorch3d. The whole module is differentiable and can be integrated into other systems for the gradient propagation.
Updates
:star2: Refactored the code to make future extension easier (other 3DMM models will be supported). The hyper-parameters are also re-organized.
:star2: Added support for monocular video reconstruction.
:star2: Added support for multi-gpu when processing long videos.
Installation
Requirements
pytorch3d It might require a specific version of Pytorch to make pytorch3d run succussfully on gpus, please follow the official instructions.
Please refer to "requirements.txt" for other dependences.
The script will extract frames, detact features and fit all frames.
Fitting a video is a bit different from fitting an image, because frames are not isolated. In this implementation, we first estimate shape and texture of the target face using some of the frames (indicated by --nframes_shape). Then we estimate the other coefficients (expression, rotatation etc.) for each frame and keep the shape and texture coefficients fixed.
For the first frame, we use much more iterations to get a good starting point. For remaining frames, each is initialized from the previous estimation.Pose regulerizers are also imposed to increase temporal consistency.
Please check the reconstructed video in the result folder.
Multi-gpu and -process support
It could take quite a while to process a long video, so multi-gpu and multi-process are also supported to accelerate fitting. To use this feature, simply run:
Here we use 4 gpus and 4 workers. We can also use more workers to assign each gpu with multiple workers. The video will be evenly split and each clip will be fit respectively.
Hyperparameters
There are bunch of parameters that might require further tuning.
The iteration numbers for non-rigid fitting "--first_nrf_iters" and "--rest_nrf_iters" affect the fitting speed a lot since we have to render the 3d mesh in each iteration. Try to change it to find a trade-off between speed and accuracy.
If the result is not good, try to play with the parameters.
Acknowledgement
The code is partially borrowed from Deep3DFaceReconstrution, which is a Tensorflow-based deep reconstruction method using CNNs. Please note that our framework does not require any pretrained deep models. We estimate the parameters directly using the landmarks and photometric loss as the supervision.
