DeepSeek Coder V2

Про DeepSeek

Засновано 2023 · Hangzhou, China

DeepSeek AI was founded in July 2023 in Hangzhou by Liang Wenfeng, who is also co-founder of the High-Flyer quantitative hedge fund. High-Flyer's GPU cluster (thousands of NVIDIA A100/H800 cards stockpiled before US export controls tightened) bootstrapped DeepSeek's training capacity. The lab is known globally for highly efficient training recipes documented in transparent technical reports. The DeepSeek-Coder line started with V1 (1.3B-33B dense models, November 2023). DeepSeek-Coder V2, released June 2024, brought MoE scaling — a 236B/21B-active model that matched or exceeded GPT-4 Turbo on code benchmarks at release. A 'Lite' 16B-active sibling was also released. DeepSeek's later DeepSeek-V3 (December 2024) and DeepSeek-R1 (January 2025) absorbed many DeepSeek-Coder design lessons. All releases use the permissive DeepSeek License with broad commercial-use rights.

Архітектура

Sparse Mixture-of-Experts Transformer for code (DeepSeekMoE + Multi-head Latent Attention)

DeepSeek-Coder V2 is a Sparse Mixture-of-Experts Transformer using the DeepSeekMoE architecture: 160 fine-grained experts plus 2 shared 'always-on' experts per MoE layer, with top-6 routing among the 160. This fine-grained + shared design (introduced in the DeepSeek-MoE paper) gives better expert specialisation than coarse 8x or 16x MoEs at similar active-parameter budgets. The model has 60 layers and 5,120 hidden size, uses Multi-head Latent Attention (MLA) for memory-efficient KV cache, RoPE position embeddings with a 128K context extension, and SwiGLU activations. The 100,000-token DeepSeek BPE tokeniser is shared across the DeepSeek family. Training began from the DeepSeek-V2 base (8.1T tokens pretraining) and added 6T more tokens of code, code-related natural language and math reasoning data. Programming language coverage is 338 languages — broader than any contemporaneous open code model. The model supports fill-in-the-middle via `<|fim_begin|>`, `<|fim_hole|>` and `<|fim_end|>` tokens. Post-training uses supervised fine-tuning plus DeepSeek's Group Relative Policy Optimisation (GRPO) RL method. A 16B-active 'Lite' variant (DeepSeek-Coder-V2-Lite) is also released. Open weights are released under the permissive DeepSeek License Agreement, which allows commercial use.

Параметри

236B total, 21B active per token (160 fine-grained experts + 2 shared, top-6 routing)

Контекст

128 000 токенів

Можливості

• 236B total / 21B active — MoE cheaper to serve than 200B+ dense alternatives
• Matched or exceeded GPT-4 Turbo on HumanEval, MBPP, LiveCodeBench at release
• Supports 338 programming languages — broader than Codestral, CodeLlama or any peer at release
• 128K context for whole-repo and large-file reasoning
• Native fill-in-the-middle (FIM) tokens for IDE completion
• Multi-head Latent Attention for memory-efficient inference
• Strong general reasoning and math from the V2 base — not just code
• Open weights under permissive DeepSeek License (commercial use allowed)
• Best for: high-quality code generation, repository-scale reasoning, polyglot codebases, self-hosted production code AI.

Навчання та ліцензія

Continued pretraining from DeepSeek-V2 base (8.1T tokens). Added 6T tokens of code-and-math-heavy data: code repositories across 338 languages (60% of the additional mix), code-related natural language (10%), math reasoning data (10%) and web data (20%). Knowledge cutoff November 2023. Post-training is supervised fine-tuning plus GRPO RL on reasoning and code benchmarks.

Ліцензія: DeepSeek License Agreement. Permissive commercial license with standard acceptable-use restrictions. No revenue threshold or separate-licence requirement — among the most liberal frontier-scale open-weights licenses.

Тестування безпеки: DeepSeek publishes a model card with limitations. The model filters politically sensitive topics consistent with Chinese regulations but is permissive on technical dual-use queries (typical for code models). No formal third-party red-team report.