CogView4 & CogView3 & CogView-3Plus

阅读中文版 日本語で読む

🤗 HuggingFace Space 🤖ModelScope Space 🛠️ZhipuAI MaaS(Faster)

👋 WeChat Community 📚 CogView3 Paper

Project Updates

• 🔥🔥 2025/03/24: We are launching CogKit, a powerful toolkit for fine-tuning and inference of the CogView4 and CogVideoX series, allowing you to fully explore our multimodal generation models.
• 2025/03/04: We've adapted and open-sourced the diffusers version of CogView-4 model, which has 6B parameters, supports native Chinese input, and Chinese text-to-image generation. You can try it online.
• 2024/10/13: We've adapted and open-sourced the diffusers version of CogView-3Plus-3B model. You can try it online.
• 2024/9/29: We've open-sourced CogView3 and CogView-3Plus-3B. CogView3 is a text-to-image system based on cascading diffusion, using a relay diffusion framework. CogView-3Plus is a series of newly developed text-to-image models based on Diffusion Transformer.

Project Plan

• [X] Diffusers workflow adaptation
• [X] Cog series fine-tuning kits (coming soon)
• [ ] ControlNet models and training code

Community Contributions

We have collected some community projects related to this repository here. These projects are maintained by community members, and we appreciate their contributions.

• ComfyUI_CogView4_Wrapper - An implementation of the CogView4 project in ComfyUI.

Model Introduction

Model Comparison

Model Name

CogView4

CogView3-Plus-3B

Resolution

        512 <= H, W <= 2048   

        H * W <= 2^{21}   

        H, W \mod 32 = 0

Inference Precision

Only supports BF16, FP32

Encoder

GLM-4-9B

T5-XXL

Prompt Language

Chinese, English

English

Prompt Length Limit

1024 Tokens

224 Tokens

Download Links

🤗 HuggingFace
🤖 ModelScope
🟣 WiseModel

🤗 HuggingFace
🤖 ModelScope
🟣 WiseModel

Memory Usage

DIT models are tested with BF16 precision and batchsize=4, with results shown in the table below:

Additionally, we recommend that your device has at least 32GB of RAM to prevent the process from being killed.

Model Metrics

We've tested on multiple benchmarks and achieved the following scores:

DPG-Bench

GenEval

T2I-CompBench

Chinese Text Accuracy Evaluation

Inference Model

Prompt Optimization

Although CogView4 series models are trained with lengthy synthetic image descriptions, we strongly recommend using a large language model to rewrite prompts before text-to-image generation, which will greatly improve generation quality.

We provide an example script. We recommend running this script to refine your prompts. Note that CogView4 and CogView3 models use different few-shot examples for prompt optimization. They need to be distinguished.

cd inference
python prompt_optimize.py --api_key "Zhipu AI API Key" --prompt {your prompt} --base_url "https://open.bigmodel.cn/api/paas/v4" --model "glm-4-plus" --cogview_version "cogview4"

Inference Model

Run the model CogView4-6B with BF16 precision:

from diffusers import CogView4Pipeline
import torch

pipe = CogView4Pipeline.from_pretrained("THUDM/CogView4-6B", torch_dtype=torch.bfloat16).to("cuda")

# Open it for reduce GPU memory usage
pipe.enable_model_cpu_offload()
pipe.vae.enable_slicing()
pipe.vae.enable_tiling()

prompt = "A vibrant cherry red sports car sits proudly under the gleaming sun, its polished exterior smooth and flawless, casting a mirror-like reflection. The car features a low, aerodynamic body, angular headlights that gaze forward like predatory eyes, and a set of black, high-gloss racing rims that contrast starkly with the red. A subtle hint of chrome embellishes the grille and exhaust, while the tinted windows suggest a luxurious and private interior. The scene conveys a sense of speed and elegance, the car appearing as if it's about to burst into a sprint along a coastal road, with the ocean's azure waves crashing in the background."
image = pipe(
    prompt=prompt,
    guidance_scale=3.5,
    num_images_per_prompt=1,
    num_inference_steps=50,
    width=1024,
    height=1024,
).images[0]

image.save("cogview4.png")

For more inference code, please check:

1. For using BNB int4 to load text encoder and complete inference code annotations, check here.
2. For using TorchAO int8 or int4 to load text encoder & transformer and complete inference code annotations, check here.
3. For setting up a gradio GUI DEMO, check here.

Fine-tuning

This repository does not contain fine-tuning code, but you can fine-tune using the following two approaches, including both LoRA and SFT:

1. CogKit, our officially maintained system-level fine-tuning framework that supports CogView4 and CogVideoX.
2. finetrainers, a low-memory solution that enables fine-tuning on a single RTX 4090.
3. If you want to train ControlNet models directly, you can refer to the training code and train your own models.

License

The code in this repository and the CogView3 models are licensed under Apache 2.0.

We welcome and appreciate your code contributions. You can view the contribution guidelines here.