Flow-Factory

Unified Online RL and Offline Fine-Tuning for Diffusion and Flow-Matching Models

🔥 News

• [2026-09-20] Qwen-Image 2.1 support! Train the unified text-to-image and ordered multi-image editing model with the GRPO + LoRA recipe. Until Qwen-Image 2.1 reaches a released Diffusers package, install the pinned submodule:

git submodule update --init
pip install -e ./diffusers
pip install -e .

• [2026-08-21] MiniMax H3 Audio-Video support! Fine-tune text-to-audio-video, first/last-frame-to-audio-video, and ordered-reference-to-audio-video workflows with GRPO + LoRA. H3 requires diffusers 0.40.0 or newer:

pip install 'diffusers>=0.40.0'
pip install -e .

• [2026-04-25] LTX-2 Audio-Video support! Generate synchronized audio-video content with RL fine-tuning through the released Diffusers API:

pip install 'diffusers>=0.40.0'

• [2026-02-01] Support for multiple Attention Backends! Attention-backend selection now lives in the unified acceleration: block (the old model.attn_backend knob was removed), where it can be combined with torch.compile and feature caching — applied in list order:

  acceleration:
    shared:
      - name: attention_backend
        params: { backend: "flash" } # Options: "native", "xformers", "flash_hub", "_flash_3_hub", "_flash_3_varlen_hub"

This experimental feature leverages diffusers's transformer.set_attention_backend. Check the official diffusers documentation for all available options. See guidance/acceleration.md for the full acceleration layer.

We recommend installing the kernels package (pip install kernels) and using flash_hub, flash_varlen_hub, _flash_3_hub, or _flash_3_varlen_hub to avoid the complexity and potential incompatibility of installing Flash-Attention directly.

📕 Table of Contents

• Supported Models
• Supported Algorithms
• Get Started
  • Installation
  • Experiment Trackers
  • Quick Start Example
• Guidance
• Dataset
  • Offline SFT and Preference Data
  • Text-to-Image & Text-to-Video
  • Image-to-Image & Image-to-Video
• Reward Model
• Acknowledgements

🤗 Supported Models

Task

Model

Model Size

Model Type

Text-to-Image

stable-diffusion-3.5-medium/large

2.5B/8.1B

sd3-5

FLUX.1-dev

13B

flux1

Z-Image-Turbo

6B

z-image

Z-Image

6B

z-image

Qwen-Image

20B

qwen-image

Qwen-Image-2512

20B

qwen-image

Image-to-Image

FLUX.1-Kontext-dev

13B

flux1-kontext

Image(s)-to-Image

Qwen-Image-Edit-2509

20B

qwen-image-edit-plus

Qwen-Image-Edit-2511

20B

qwen-image-edit-plus

Text-to-Image & Image(s)-to-Image

FLUX.2-dev

32B

flux2

FLUX.2-klein-4B

4B

flux2-klein

FLUX.2-klein-9B

9B

flux2-klein

FLUX.2-klein-base-4B

4B

flux2-klein

FLUX.2-klein-base-9B

9B

flux2-klein

BAGEL-7B-MoT

14B

bagel

SenseNova-U1 1.0

16B

sensenova

SenseNova-U1 1.5

16B

sensenova

Qwen-Image 2.1

7B

qwen-image-2.1

Text-to-Video

Wan2.1-T2V-1.3B

1.3B

wan2_t2v

Wan2.1-T2V-14B

14B

wan2_t2v

Wan2.2-TI2V-5B

5B

wan2_t2v

Wan2.2-T2V-A14B

A14B

wan2_t2v

Image-to-Video

Wan2.1-I2V-14B-480P

14B

wan2_i2v

Wan2.1-I2V-14B-720P

14B

wan2_i2v

Wan2.2-TI2V-5B

5B

wan2_i2v

Wan2.2-I2V-A14B

A14B

wan2_i2v

First/Last-Frame-to-Video

Wan2.1-FLF2V-14B-720P

14B

wan2_i2v

Text-to-Audio-Video

LTX-2

19B

ltx2_t2av

LTX-2.3 (Diffusers)

22B

ltx2_t2av

Image-to-Audio-Video

LTX-2

19B

ltx2_i2av

LTX-2.3 (Diffusers)

22B

ltx2_i2av

Text-to-Audio-Video

MiniMax H3 T2VA

33B

minimax-h3-t2va

First/Last-Frame-to-Audio-Video

MiniMax H3 FL2VA

33B

minimax-h3-fl2va

Ordered-Reference-to-Audio-Video

MiniMax H3 Ref2VA

33B

minimax-h3-ref2va

To support new models, see Guidance/New Model.

Offline output support: SFT and offline DPO currently support sd3-5, flux1, flux1-kontext, flux2, flux2-klein, qwen-image, qwen-image-edit-plus, qwen-image-2.1, z-image, bagel, sensenova, wan2_t2v, wan2_i2v, ltx2_t2av, ltx2_i2av, and all MiniMax H3 workflows. Video/audio targets are encoded on demand and are never written to the preprocessing cache. Conditioned adapters prepare one immutable condition state per batch; offline DPO shares that exact realization across chosen and rejected arms. See the offline model matrix.

MiniMax H3 status: T2VA, FL2VA, and Ref2VA completed all 36 real-weight smoke cells in the documented matrix: three workflows x DDP/DeepSpeed ZeRO-2/FSDP2 x GRPO/SFT/offline DPO/TDM. The FL2VA first-plus-last SFT/offline-DPO gate also passed. The T2VA native-quality FSDP2 path has separate initialization, checkpoint, decode, and evaluation coverage. These results do not claim a completed long-run reward trend, convergence, or numerical parity. H3 requires B=1, has no CFG, uses neutral guidance 1.0, and keeps separate video/audio trajectories. Video uses shift 12, audio uses shift 3, and the model predicts data-ward velocity. num_inference_steps=N means N transitions and N + 1 states.

💻 Supported Algorithms

See Algorithm Guidance for more information.

Models and algorithms are decoupled at the framework interface. The documented ten-mode real-weight smoke matrix completed all 120 model/backend/algorithm cells. Combinations outside that matrix still require separate execution evidence, and smoke completion is not a claim of reward improvement.

💾 Hardware Requirements

🚀 Get Started

Installation

git clone https://github.com/X-GenGroup/Flow-Factory.git
cd Flow-Factory
pip install -e .

Flow-Factory requires Python 3.10 or newer and PyTorch 2.10 or newer. PyTorch 2.10 includes the native torch.optim.Muon API used by Muon optimizer configs.

Optional dependencies, such as deepspeed, are also available. Install them with:

pip install -e .[deepspeed]

Note: The Bagel adapter requires flash-attn (>= 2.5.8) and opencv-python. Install them with pip install -e .[bagel] (the [bagel] extra is intentionally not part of [all] because flash-attn is heavy to build).

Dependency: MiniMax H3 and LTX2 require the released diffusers>=0.40.0 API. Qwen-Image 2.1 currently requires the bundled diffusers submodule; run git submodule update --init && pip install -e ./diffusers before installing Flow-Factory. Its initial adapter deliberately uses batch size 1 and disables the model's prefix KV cache so online rollout and gradient replay follow the same numerical path. PyAV >=17.0.0 decodes ordered video/audio references and target media. TorchAudio 2.10 delegates audio loading and saving to TorchCodec, which also requires FFmpeg shared libraries. The CUDA image installs those system libraries automatically.

A CUDA training image (Python 3.12, uv-based install, PyTorch 2.10 + cu129, deepspeed, wandb, released diffusers) is defined under docker/docker-cuda/. See docker/README.md for build and run instructions (including linux/amd64 on Apple Silicon).

Experiment Trackers

To use Weights & Biases or SwanLab to log experimental results, install extra dependencies via pip install -e .[wandb] or pip install -e .[swanlab].

After installation, set corresponding arguments in the config file:

run_name: null  # Run name (auto: {model_type}_{finetune_type}_{trainer_type}_{timestamp})
project: "Flow-Factory"  # Project name for logging
logging_backend: "wandb"  # Options: wandb, swanlab, tensorboard, none

These trackers allow you to visualize both training samples and metric curves online:

Quick Start Example

Start training with the following simple command:

ff-train examples/grpo/lora/flux1/default.yaml

Offline examples use strict V2 manifests and require no training reward model:

ff-train examples/sft/lora/sd3_5/default.yaml
ff-train examples/offline_dpo/lora/sd3_5/default.yaml
ff-train examples/sft/lora/wan2_t2v/wan22_ti2v5b.yaml

📖 Guidance

We provide a set of guidance documents to help you understand the framework and extend it. For a comprehensive understanding of the framework's design and motivation, refer to our technique report.

📊 Dataset

The unified structure of dataset is:

|---- dataset
|----|--- train.txt / train.jsonl
|----|--- test.txt / test.jsonl (optional)
|----|--- images (optional)
|----|---| image1.png
|----|---| ...
|----|--- videos (optional)
|----|---| video1.mp4
|----|---| ...

Offline SFT and Preference Data

SFT and offline DPO use strict JSONL with schema_version: 2. Public media objects use type as their sole discriminator:

{"schema_version":2,"input":{"prompt":"A clean poster.","media":[]},"supervision":{"type":"demonstration","target":{"media":[{"type":"image","path":"targets/poster.png"}]}},"metadata":{}}
{"schema_version":2,"input":{"prompt":"A clean poster.","media":[]},"supervision":{"type":"preference","chosen":{"media":[{"type":"image","path":"pairs/chosen.png"}]},"rejected":{"media":[{"type":"image","path":"pairs/rejected.png"}]}},"metadata":{}}
{"schema_version":2,"input":{"prompt":"Animate toward this ending.","media":[{"type":"image","path":"conditions/end.png","slot":"last_frame"}]},"supervision":{"type":"demonstration","target":{"media":[{"type":"video","path":"targets/story.mp4","fps":24.0},{"type":"audio","path":"targets/story.wav","sample_rate":32000}]}},"metadata":{}}

The optional input-only slot field binds sparse conditions to adapter-declared semantic arguments. Unslotted media fills remaining slots positionally; supervision outputs reject slots.

The checked-in SFT demonstration fixture and offline-DPO preference fixture provide minimal examples under the repository's canonical dataset/ root.

Prompt and input-condition encodings are cached. Target, chosen, and rejected media are decoded and encoded on the fly; their VAE latents are never stored in the preprocessing cache. One offline epoch is one complete dataloader traversal sharded by PyTorch's official DistributedSampler. See the dataset guide for the full schema and cadence rules, and the completed GPU validation matrix for the 120 main jobs and 24 additional dynamic gates. The offline smoke builder reconstructs independent SFT and offline-DPO mini datasets for every currently implemented image, video, and audio-video profile.

Text-to-Image & Text-to-Video

For text-to-image and text-to-video tasks, the only required input is the prompt in plain text format. Use train.txt and test.txt (optional) with following format:

A hill in a sunset.
An astronaut riding a horse on Mars.

Example: dataset/pickscore

Each line represents a single text prompt. Alternatively, you can use train.jsonl and test.jsonl in the following format:

{"prompt": "A hill in a sunset."}
{"prompt": "An astronaut riding a horse on Mars."}

Example: dataset/t2is

negative_prompt is also supported:

{"prompt": "A hill in a sunset.", "negative_prompt": "low quality, blurry, distorted, poorly drawn"}
{"prompt": "An astronaut riding a horse on Mars.", "negative_prompt": "low quality, blurry, distorted, poorly drawn"}

Example: dataset/t2is_neg

Image-to-Image & Image-to-Video

For tasks involving conditioning images, use train.jsonl and test.jsonl in the following format:

{"prompt": "A hill in a sunset.", "image": "path/to/image1.png"}
{"prompt": "An astronaut riding a horse on Mars.", "image": "path/to/image2/png"}

Example: dataset/sharegpt4o_image_mini

The default root directory for images is dataset_dir/images, and for videos, it is dataset_dir/videos. You can override these locations by setting the image_dir and video_dir variables in the config file:

data:
    dataset_dir: "path/to/dataset"
    image_dir: "path/to/image_dir" # (default to "{dataset_dir}/images")
    video_dir: "path/to/video_dir" # (default to "{dataset_dir}/videos")

For models such as FLUX.2-dev, Qwen-Image-Edit-2511, BAGEL-7B-MoT, and SenseNova-U1 that accept multiple conditioning images, use the images key with an ordered list of image paths:

{"prompt": "A hill in a sunset.", "images": ["path/to/condition_image_1_1.png", "path/to/condition_image_1_2.png"]}
{"prompt": "An astronaut riding a horse on Mars.", "images": ["path/to/condition_image_2_1.png", "path/to/condition_image_2_2.png"]}

💯 Reward Model

Flow-Factory provides a flexible reward model system that supports both built-in and custom reward models for reinforcement learning.

Reward Model Types

Flow-Factory supports two types of reward models:

• Pointwise Reward: Computes independent scores for each sample (e.g., aesthetic quality, text-image alignment).
• Pairwise Reward: Computes rewards based on the pairwise comparison within the group. This is a special case of the following Groupwise Reward.
• Groupwise Reward: Computes rewards that requires the all samples in a group (e.g., ranking-based score or pairwise comparison).

Built-in Reward Models

The following reward models are pre-registered and ready to use:

GenEval requires extra dependencies (mmcv, mmdet, open_clip). Install with: bash scripts/install_geneval_deps.sh (Python 3.10 or newer). See guidance/rewards.md for dataset format.

VLM-as-Judge (remote vLLM / OpenAI-style HTTP) is covered in guidance/rewards.md#vlm-as-judge (vllm_evaluate, Rational Rewards, qwen_image_bench, async tips). For RationalRewards specifically, serve the judge with scripts/start_vllm_rational_reward.sh and set YAML api_base_url / vlm_model to match --served-model-name (defaults: RationalRewards-8B-T2I / RationalRewards-8B-Edit). For Qwen-Image-Bench, use scripts/start_vllm_qwen_image_bench.sh and build the dataset with python dataset/qwen_image_bench/prepare.py.

Using Built-in Reward Models

Simply specify the reward model name in your config file:

rewards:
  name: "aesthetic" # Alias for this reward model
  reward_model: "PickScore" # Reward model type or a path like 'my_package.rewards.CustomReward'
  batch_size: 16
  device: "cuda"
  dtype: bfloat16

Refer to Rewards Guidance for more information about advanced usage, such as creating a custom reward model.

🤗 Acknowledgements

This repository is based on diffusers, accelerate and peft. We thank them for their contributions to the community!!!

📝 Citation

If you find Flow-Factory useful in your research, please consider citing our paper:

@article{ping2026flowfactory,
  title={Flow-Factory: A Unified Framework for Reinforcement Learning in Flow-Matching Models}, 
  author={Bowen Ping and Chengyou Jia and Minnan Luo and Hangwei Qian and Ivor Tsang},
  journal={arXiv preprint arXiv:2602.12529},
  year={2026},
  url={https://arxiv.org/abs/2602.12529}, 
}