DeepSeek V3

About DeepSeek

Founded 2023 · Hangzhou, China

DeepSeek AI was founded in July 2023 in Hangzhou by Liang Wenfeng, who is also co-founder of the High-Flyer quantitative hedge fund. High-Flyer's GPU cluster (thousands of NVIDIA A100/H800 cards stockpiled before US export controls tightened) bootstrapped DeepSeek's training capacity. The lab gained global attention for highly efficient training recipes documented in transparent technical reports. Notable releases include DeepSeek Coder (Nov 2023), DeepSeek LLM 67B (Jan 2024), DeepSeekMath with GRPO reinforcement learning (Feb 2024), DeepSeek V2 introducing Multi-head Latent Attention and DeepSeekMoE (May 2024), DeepSeek V3 in December 2024 and DeepSeek R1 in January 2025. The V3/R1 releases triggered global discussion when DeepSeek reported that V3 was trained for approximately $5.6M of GPU-hour cost on 2.788M H800 GPU-hours, ten or more times cheaper than comparable Western frontier runs. All models are released under MIT license. The company is privately funded by High-Flyer rather than venture capital and employs roughly 200 researchers, mostly recent PhDs from Chinese universities.

Architecture

Sparse Mixture-of-Experts Transformer (DeepSeekMoE + Multi-head Latent Attention)

DeepSeek V3 was released on 26 December 2024 with weights under MIT license. It is a Sparse Mixture-of-Experts Transformer with 671 billion total parameters and 37 billion active per token. The architecture combines DeepSeekMoE (fine-grained experts with shared experts for load balancing without auxiliary loss) and Multi-head Latent Attention (MLA), a low-rank KV-cache compression technique introduced in V2 that drastically reduces memory bandwidth during inference. V3 was pretrained on 14.8 trillion high-quality tokens spanning multilingual web text, code, books and scientific papers, using a total compute budget of 2.788 million H800 GPU-hours, which DeepSeek reports as approximately $5.576M at $2/GPU-hour. The training run introduced multi-token prediction (MTP) as an auxiliary objective and FP8 mixed-precision training with custom CUDA kernels for the MoE routing. Post-training included supervised fine-tuning on 1.5M curated examples plus a reinforcement learning stage using GRPO. V3 achieves performance competitive with GPT-4o and Claude 3.5 Sonnet on most text and code benchmarks while costing approximately 1/10th to operate, making it the highest-performing open-weight non-reasoning model at launch.

Parameters

671B total, 37B active per token

Context

128,000 tokens

Capabilities

• 671B-parameter MoE with 37B active per token
• 128K context window
• Pretrained on 14.8T tokens for ~$5.6M of compute
• DeepSeekMoE routing without auxiliary loss
• Multi-head Latent Attention for memory-efficient inference
• FP8 mixed-precision training with custom kernels
• Multi-token prediction (MTP) auxiliary objective
• Strong code generation on HumanEval, MBPP, LiveCodeBench
• Open weights under MIT license
• Compatible with vLLM, SGLang, llama.cpp, HuggingFace
• Best for: cost-efficient open-weight chat, coding, on-prem enterprise, research on MoE.

Training & license

Pretrained on 14.8 trillion tokens of curated multilingual web text, code repositories, books and scientific papers. Knowledge cutoff is approximately mid-2024. Post-training uses 1.5M-example SFT followed by GRPO reinforcement learning on preference and verifiable-reward data.

License: MIT license for model weights, code and tokenizer. Commercial use permitted without restrictions.

Safety testing: Limited published safety evaluation. The model filters politically sensitive topics in line with Chinese regulations but otherwise has a relatively low refusal rate on technical dual-use queries.