PAIR Diffusion: A Comprehensive Multimodal Object-Level Image Editor [ ] [ ] [ ] [ ] Vidit Goel 1 , Elia Peruzzo 1,2 , Yifan Jiang 3 , Dejia Xu 3 , Xingqian Xu 3 , Nicu Sebe 2 , Trevor Darrell 4 , Zhangyang Wang 1,3 and Humphrey Shi 1,5,6 Features All the operations below can be performed at an object level. Our framework is general and can be applied to any diffusion model. Multimodal control using reference images and text News [30/12/2023] Models and code released
. [10/11/2023] New and improved method and models 🚀🚀. [04/09/2023] Inference code released [04/07/2023] Demo relased on 🤗Huggingface space! [03/30/2023] Paper released on arXiv Results Given below are results for appearace editing using our method on SDv1. 5 Object Level Image Editing Appearance Editing Free form Shape Editing and Adding Object For more results please refer to our project page and paper. Requirements Setup the conda environment using the command below. We use Oneformer
to get segmentation maps during inference, please setup environment for Oneformer following the repo Inference To run the model launch the gradio demo using the command below. It will download the required models as well. Pretrained Models We applied PAIR Diffusion on SDv1. 5 and uses COCO-Stuff dataset for finetuning the model. The model card can be downloaded from here BibTeX If you use our work in your research, please cite our publication:
Given below are results for appearace editing using our method on SDv1.5
Object Level Image Editing
Appearance Editing
Free form Shape Editing and Adding Object
For more results please refer to our project page and paper.
Requirements
Setup the conda environment using the command below. We use Oneformer to get segmentation maps during inference, please setup environment for Oneformer following the repo
To run the model launch the gradio demo using the command below. It will download the required models as well.
python gradio_demo.py
Pretrained Models
We applied PAIR Diffusion on SDv1.5 and uses COCO-Stuff dataset for finetuning the model. The model card can be downloaded from here
BibTeX
If you use our work in your research, please cite our publication:
@article{goel2023pair,
title={PAIR-Diffusion: Object-Level Image Editing with Structure-and-Appearance Paired Diffusion Models},
author={Goel, Vidit and Peruzzo, Elia and Jiang, Yifan and Xu, Dejia and Sebe, Nicu and Darrell, Trevor and
Wang, Zhangyang and Shi, Humphrey},
journal={arXiv preprint arXiv:2303.17546},
year={2023}
}
