In-Context LoRA (IC-LoRA)

🔥 Latest News!

• [2024-12-17] 🚀 We are excited to release IDEA-Bench, a comprehensive benchmark designed to assess the zero-shot task generalization abilities of generative models. The benchmark includes 100 real-world design tasks across 275 unique cases. Despite its general-purpose focus, the top-performing model, EMU2, achieves a score of only 6.81 out of 100, highlighting the current challenges in this domain. Explore the benchmark and challenge the limits of model performance!
• [2024-11-16] 🌟 The community continues to innovate with IC-LoRA! Exciting projects include models, ComfyUI nodes and workflows for Virtual Try-on, Product Design, Object Mitigation, Role Play, and more. Explore their creations in Community Creations Using IC-LoRA. Huge thanks to all contributors for their incredible efforts!
• [2024-11-07] 🚀 We have released 10 pretrained models for In-Context LoRA, covering diverse tasks such as Film Storyboard Generation, Visual Identity Design, and Visual Effects. See MODEL ZOO for details. We also provide an example workflow for ComfyUI.
• [2024-11-01] 📂 Data and training configurations for In-Context LoRA are now available!
• [2024-10-31] 📜 Our latest paper, In-Context LoRA, introduces a flexible framework adaptable to a wide range of tasks.
• [2024-10-19] 🎨 We released the paper Group Diffusion Transformers, the predecessor of In-Context LoRA, offering zero-shot support for 30 visual generation tasks.
• [2024-4-18] 💻 We release the code and models for FlashFace, a precursor to Group Diffusion Transformers, verifies attention token concatenation for customized generation scenarios.

Welcome to the official repository of In-Context LoRA for Diffusion Transformers (Paper and Project Page).

Community Creations Using IC-LoRA

We are thrilled to showcase the community's innovative projects leveraging In-Context LoRA (IC-LoRA). If you have additional recommendations or projects to share, please don't hesitate to send a Pull Request!

Project Name	Type	Supported Tasks	Sample Results
1. Comfyui_Object_Migration	ComfyUI Node & Workflow & LoRA Model	Clothing Migration, Cartoon Clothing to Realism, and More
2. Flux Simple Try On - In Context Lora	LoRA Model & ComfyUI Workflow	Virtual Try-on
3. Flux In Context - visual identity Lora in Comfy	ComfyUI Workflow	Visual Identity Transfer
4. Workflows Flux In Context Lora For Product Design	ComfyUI Workflow	Product Design, Role Play, and More
5. Flux Product Design - In Context Lora	LoRA Model & ComfyUI Workflow	Product Design
6. In Context lora + Character story generator + flux+ shichen	ComfyUI Workflow	Character Movie Story Generator
7. In- Context-Lora｜Cute 4koma 可爱四格漫画	LoRA Model & ComfyUI Workflow	Comic Strip Generation
8. Creative Effects & Design LoRA Pack (In-Context LORA)	LoRA Model & ComfyUI Workflow	Movie-Shot Generation and More

We extend our heartfelt thanks to all contributors for their exceptional work in advancing the IC-LoRA ecosystem.

Key Idea

The core concept of IC-LoRA is to concatenate both condition and target images into a single composite image while using Natural Language to define the task. This approach enables seamless adaptation to a wide range of applications.

Features

• Task-Agnostic Framework: IC-LoRA serves as a general framework, but it requires task-specific fine-tuning for diverse applications.
• Customizable Image-Set Generation: You can fine-tune text-to-image models to generate image sets with customizable intrinsic relationships.
• Condition on Image-Set: You can also condition the generation of a set of images on another set of images, enabling a wide range of controllable generation applications.

For more detailed information and examples, please read our Paper or visit our Project Page.

Getting Started

You can directly use the open-source AI-Toolkit to train IC-LoRA models. We have provided sample training data with a configuration file in this repo:

• Configuration File: config/movie-shots.yml (place it in the config/ directory of AI-Toolkit)
• Sample Training Data: data/movie-shots.zip (extract it to data/movie-shots of AI-Toolkit)

After installing the necessary dependencies and setting up AI-Toolkit, you can start training by running:

python run.py config/movie-shots.yml

The training runs on a single GPU with at least 24GB of memory (adjust the resolution parameter in config/movie-shots.yml for different GPU memory limits). The training should complete in a few hours.

Prompt for Multi-Scene Image Captioning

As a reference, we provide an example prompt used to generate captions for multi-scene images:

Create a short description of this three-scene image featuring movie shots, beginning with the prefix [MOVIE-SHOTS] for the entire caption, followed by an overall summary of the image. Each scene detail should flow within the same sentence, with specific markers [SCENE-1], [SCENE-2], [SCENE-3], indicating the start of each scene’s description. Name the role(s) with random name(s) if necessary, and wrap the name(s) with "<" and ">". Ensure the entire description is cohesive, flows as one sentence, and remains within 512 words.

MODEL ZOO

Below lists 10 In-Context LoRA models and their recommend settings. We provide an example workflow for ComfyUI.

License

This repository uses FLUX as the base model. Users must comply with FLUX's license when using this code. Please refer to FLUX's License for more details.

DISCLAIMER: Please be aware that the training data provided in this repository may contain copyrighted material. The open-source data is intended for reference and educational purposes only. If you plan to use this data for commercial purposes, you are responsible for obtaining the necessary permissions and ensuring compliance with all applicable copyright laws and regulations.

Citation

If you find this work useful in your research, please consider citing:

@article{lhhuang2024iclora,
  title={In-Context LoRA for Diffusion Transformers},
  author={Huang, Lianghua and Wang, Wei and Wu, Zhi-Fan and Shi, Yupeng and Dou, Huanzhang and Liang, Chen and Feng, Yutong and Liu, Yu and Zhou, Jingren},
  journal={arXiv preprint arxiv:2410.23775},
  year={2024}
}

@article{lhhuang2024groupdiffusion,
  title={Group Diffusion Transformers are Unsupervised Multitask Learners},
  author={Huang, Lianghua and Wang, Wei and Wu, Zhi-Fan and Dou, Huanzhang and Shi, Yupeng and Feng, Yutong and Liang, Chen and Liu, Yu and Zhou, Jingren},
  journal={arXiv preprint arxiv:2410.15027},
  year={2024}
}