X-Portrait

X-Portrait: Expressive Portrait Animation with Hierarchical Motion Attention

            [You Xie](https://scholar.google.com/citations?user=FV0eXhQAAAAJ&hl=en),
            [Hongyi Xu](https://hongyixu37.github.io/homepage/),
            [Guoxian Song](https://guoxiansong.github.io/homepage/index.html),
            [Chao Wang](https://chaowang.info/),
            [Yichun Shi](https://seasonsh.github.io/),
            [Linjie Luo](http://linjieluo.com/)
  

   ByteDance Inc. 
  

  

    [![Paper PDF](https://img.shields.io/badge/arXiv-X--Portrait-red)](https://arxiv.org/abs/2403.15931)
    [![Project Page](https://img.shields.io/badge/Project_Page-X--Portrait-green)](https://byteaigc.github.io/x-portrait/)
    [![Youtube](https://img.shields.io/badge/YouTube-X--Portrait-rgb(255, 0, 0))](https://youtu.be/VGxt5XghRdw)
  










   ![](assets/teaser/teaser.png) 

This repository contains the video generation code of SIGGRAPH 2024 paper X-Portrait.

Installation

Note: Python 3.9 and Cuda 11.8 are required.

bash env_install.sh

Model

Please download pre-trained model from here, and save it under "checkpoint/"

Testing

bash scripts/test_xportrait.sh

parameters:
model_config: config file of the corresponding model
output_dir: output path for generated video
source_image: path of source image
driving_video: path of driving video
best_frame: specify the frame index in the driving video where the head pose best matches the source image (note: precision of best_frame index might affect the final quality)
out_frames: number of generation frames
num_mix: number of overlapping frames when applying prompt travelling during inference
ddim_steps: number of inference steps (e.g., 30 steps for ddim)

Performance Boost

efficiency: Our model is compatible with LCM LoRA (https://huggingface.co/latent-consistency/lcm-lora-sdv1-5), which helps reduce the number of inference steps.
expressiveness: Expressiveness of the results could be boosted if results of other face reenactment approaches, e.g., face vid2vid, could be provided via parameter "--initial_facevid2vid_results".

🎓 Citation

If you find this codebase useful for your research, please use the following entry.

@inproceedings{xie2024x,
  title={X-Portrait: Expressive Portrait Animation with Hierarchical Motion Attention},
  author={Xie, You and Xu, Hongyi and Song, Guoxian and Wang, Chao and Shi, Yichun and Luo, Linjie},
  journal={arXiv preprint arXiv:2403.15931},
  year={2024}
}