HyperLoRA: Parameter-Efficient Adaptive Generation for Portrait Synthesis

[Mengtian Li](https://lemonsky1995.github.io/)*,
[Jinshu Chen](https://openreview.net/profile?id=~Jinshu_Chen2)*,
[Wanquan Feng](https://wanquanf.github.io/)*✉,
[Bingchuan Li](https://scholar.google.com/citations?user=ac5Se6QAAAAJ),
[Fei Dai](https://openreview.net/profile?id=~Fei_Dai1),
[Songtao Zhao](https://openreview.net/profile?id=~Songtao_Zhao1),
[Qian He](https://scholar.google.com/citations?user=9rWWCgUAAAAJ)
  

Intelligent Creation, ByteDance






[[Project Page](https://lemonsky1995.github.io/hyperlora/)]
[[ArXiv](https://arxiv.org/abs/2503.16944)]

Abstract

Personalized portrait synthesis, essential in domains like social entertainment, has recently made significant progress. Person-wise fine-tuning based methods, such as LoRA and DreamBooth, can produce photorealistic outputs but need training on individual samples, consuming time and resources and posing an unstable risk. Adapter based techniques such as IP-Adapter freeze the foundational model parameters and employ a plug-in architecture to enable zero-shot inference, but they often exhibit a lack of naturalness and authenticity, which are not to be overlooked in portrait synthesis tasks. In this paper, we introduce a parameter-efficient adaptive generation method, namely HyperLoRA, that uses an adaptive plug-in network to generate LoRA weights, merging the superior performance of LoRA with the zero-shot capability of adapter scheme. Through our carefully designed network structure and training strategy, we achieve zero-shot personalized portrait generation (supporting both single and multiple image inputs) with high photorealism, fidelity, and editability.

Overview

We explicitly decompose the HyperLoRA into a Hyper ID-LoRA and a Hyper Base-LoRA. The former is designed to learn ID information while the latter is expected to fit others, e.g. background and clothing. Such a design helps to prevent irreverent features leaking to ID-LoRA. During the training, we fix the weights of the pretrained SDXL base model and encoders, allowing only HyperLoRA modules updated by Backpropagation. At the inference stage, the Hyper ID-LoRA integrated into SDXL generates personalized images while the Hyper Base-LoRA is optional.

Usage

Create Model Folders

Navigate to your ComfyUI's models folder and create the following subfolders:

models/
├── hyper_lora/
│   ├── clip_processor/    # Directory for CLIP processor files
│   │   └── clip_vit_large_14_processor/     # Specific CLIP Processor
│   │       └── preprocessor_config.json
│   ├── clip_vit/          # Directory for CLIP ViT model files
│   │   └── clip_vit_large_14/               # Specific CLIP ViT model
│   │       ├── config.json
│   │       └── model.safetensors
│   └── hyper_lora/        # Directory for HyperLoRA model files
│       ├── sdxl_hyper_id_lora_v1_edit/      # HyperLoRA for editability
│       │   ├── hyper_lora_modules.json
│       │   ├── hyper_lora_modules.safetensors
│       │   ├── id_projector.safetensors
│       │   └── resampler.safetensors
│       └── sdxl_hyper_id_lora_v1_fidelity/  # HyperLoRA for fidelity
│           └── Same as above
└── insightface/
    └── models/            # Directory for InsightFace model files

Download Models

First, download the CLIP processor to models/hyper_lora/clip_processor/clip_vit_large_14_processor

Second, download the CLIP ViT model and config to models/hyper_lora/clip_vit/clip_vit_large_14

Then, download the antelopev2 and unzip it to models/insightface/models

Finally, download the HyperLoRA model to models/hyper_lora/hyper_lora

We have released two versions of HyperLoRA. sdxl_hyper_id_lora_v1_fidelity offers better fidelity, while sdxl_hyper_id_lora_v1_edit provides better editability. Note that we trained for some additional steps using FP16 precision and distilled the 0.25x Base LoRA into the ID LoRA, which further reduced the model size and the GPU memory usage.

Example Workflows

The workflows depend on ComfyUI_ADV_CLIP_emb and ComfyUI-Impact-Pack plugins.

Name

Workflow

Result

T2I

HyperLoRA-T2I.json

T2I + FaceDetailer🌟

HyperLoRA-T2I-FaceDetailer.json

T2I + ControlNet

HyperLoRA-T2I-ControlNet.json

T2I + InstantID🌟

HyperLoRA-T2I-InstantID.json

T2I + ConceptSlider

HyperLoRA-T2I-ConceptSlider.json

🌟 Recommended workflow

We fixed the random seed in all workflows so that the results can be fully reproduced. All the input images required by these workflows can be found in the assets folder.

Non-ComfyUI User

We also provide a standalone script to generate ID LoRAs, which works without ComfyUI. Please create model folders, download models, and run the following command.

python3 standalone/generate_loras.py \
    --models_dir  \
    --indir  \
    --outdir  \
   [--model {sdxl_hyper_id_lora_v1_fidelity|sdxl_hyper_id_lora_v1_edit}] \
   [--dtype {fp16|bf16|fp32}]

Tips

IMPORTANT:

• Please confirm that all model and config files are placed in correct directories.
• The text prompt should start with the LoRA trigger words fcsks fxhks fhyks, to enable LoRA.
• It is recommended to set stop_at_clip_layer to -2 to align with our training config.

HyperLoRA is not compatible with all SDXL base models. We have tested that it is compatible with LEOSAM's HelloWorld XL 3.0, CyberRealistic XL v1.1, and RealVisXL v4.0, but incompatible with ArienMixXL v4.0. Among them, RealVisXL v4.0 may achieve the best results.

LEOSAM 😊

CyberRealistic 😊

RealVis 😊

ArienMix 😭

The LoRA weight are usually around 0.75 to 0.85. If the prompt is very long, you can appropriately increase the weight (as well as the weight of trigger words). If better editability is required, you can appropriately decrease the weight.

Sunglasses

Baby

Sketch

Sand Sculpture

Due to the limited resolution of the trained faces, it is recommended to use a FaceDetailer plugin to repair small faces or use ControlNet to increase stability. In addition, combining HyperLoRA with the ControlNet of InstantID can usually further improve the facial similarity.

When creating a Concept Slider using HyperLoRA, the paired images should ensure that only the edited attribute changes, while other attributes remain fixed (especially the ID and pose). Therefore, the best way is to render an animatable 3D human head.

Furthermore, HyperLoRA can also provide a good initialization for LoRA training. You can obtain a better ID LoRA with only about 50 steps of finetuning.

Disclaimer and Licenses

Our code is under the GPL 3.0 license. You can use, modify, and distribute it, with derivative works also under GPL 3.0. Our models follow the CC BY-NC 4.0 license, allowing non-commercial sharing and adaptation with proper credit. Third-party models like SDXL, CLIP, and InsightFace are subject to their original licenses.

This project aims to advance the field of generative AI. We disclaim any liability for misuse or unauthorized use of the code, models, or related materials. You must comply with all relevant licenses and laws when using this plugin.

Citation

If you find HyperLoRA useful in your research, please kindly cite our paper:

@article{li2025hyperlora,
    title={HyperLoRA: Parameter-Efficient Adaptive Generation for Portrait Synthesis}, 
    author={Mengtian Li and Jinshu Chen and Wanquan Feng and Bingchuan Li and Fei Dai and Songtao Zhao and Qian He},
    journal={arXiv preprint arXiv:2503.16944},
    year={2025}
}