CogVideoX-5B (open)

Over THUDM (Tsinghua University KEG Lab) / Zhipu AI

Opgericht 2019 · Beijing, China

THUDM (the Knowledge Engineering Group at Tsinghua University) and its commercial arm Zhipu AI are among China's leading open-source generative-AI research groups. The lab is best known for the GLM language-model family (GLM-130B, ChatGLM) and the CogVLM/CogVideo vision projects. CogVideo was introduced in 2022 as one of the first publicly released 9B-parameter text-to-video transformers, followed by CogVideoX in August 2024, which released 2B and 5B variants under an open-weight licence. Zhipu AI raised more than $1.5B by 2025 and is one of the four 'AI tigers' of China alongside Moonshot, MiniMax and Baichuan. CogVideoX is widely adopted as a research and fine-tuning base because of its permissive licence and reproducible training recipe.

Architectuur

Diffusion Transformer (DiT) with expert Transformer blocks and 3D causal VAE

CogVideoX is a latent video diffusion model built around a 3D causal Variational Autoencoder that compresses video into a compact latent grid across both spatial and temporal axes. On top of this latent space, an Expert Transformer (a diffusion transformer with separate text and video expert streams sharing self-attention) jointly denoises text-conditioned latent video at multiple resolutions. The architecture employs 3D Rotary Position Embeddings (3D-RoPE) for spatio-temporal positions, an Adaptive LayerNorm controlled by the diffusion timestep, and Flow Matching with v-prediction as the training objective. Training proceeds in stages: first low-resolution images, then short low-resolution videos, then high-resolution video up to 720x480 at 8 fps for 6 seconds. The team curated a large filtered video corpus with dense bilingual captions produced by a fine-tuned vision-language model. CogVideoX-5B supports text-to-video and an image-to-video variant (CogVideoX-5B-I2V).

Parameters

5 billion (also 2B variant)

Context

226 tokens

Mogelijkheden

• Open-weight 5B text-to-video model (Apache 2.0-style permissive licence on weights)
• Generates 6-second clips at 720x480 / 8 fps natively (~49 frames)
• Image-to-video variant CogVideoX-5B-I2V for conditioning on a first frame
• Strong prompt adherence on complex compositions and motion verbs
• Bilingual English/Chinese prompting
• Runs on a single 24-48 GB consumer-class GPU with optimisations (CPU offload, INT8)
• Fine-tunable with LoRA and full-parameter fine-tuning
• Frequently extended by community for longer durations via temporal tiling
• Best for: research, open-source pipelines, custom fine-tunes, on-prem video generation.

Training & licentie

Trained on a curated multi-million-clip video corpus with bilingual dense captions generated by a fine-tuned video captioning model. Data is heavily filtered for aesthetic quality, motion coherence and caption alignment. Exact token / clip counts are reported in the paper.

Licentie: Open weights under the CogVideoX Model Licence (free for research and commercial use with attribution).

Veiligheidstests: Released with safety filtering on training data and recommended NSFW classifiers for downstream deployment; no formal RSP-style policy.