Llama 3.3 70B

Про Meta AI

Засновано 2013 · Menlo Park, USA

Meta AI (originally Facebook AI Research, FAIR) was founded in December 2013 by Mark Zuckerberg with Yann LeCun as its first director. The lab is now part of Meta Platforms and houses several thousand researchers across Menlo Park, New York, Paris, Montreal, Seattle and Tel Aviv. FAIR has authored landmark papers including PyTorch (2017), Detectron, fastText, RoBERTa, the original LLaMA paper (Feb 2023), Llama 2 (Jul 2023), Llama 3 (Apr 2024), Llama 3.1 (Jul 2024, including the 405B flagship), Llama 3.2 (Sep 2024, multimodal and on-device sizes), Llama 3.3 (Dec 2024) and Llama 4 (Apr 2025). Meta's open-weights strategy under the Llama Community License has made Llama by far the most widely deployed open-weight family with over 700M cumulative downloads. Yann LeCun, Joelle Pineau and Ahmad Al-Dahle lead the GenAI organisation that productises the work. Beyond Llama, Meta ships SeamlessM4T for speech, Segment Anything for vision and the Meta AI consumer assistant across WhatsApp, Instagram, Messenger and Meta.ai. Meta is also the largest user of NVIDIA H100 GPUs in industry, with reported cluster sizes above 350,000 H100-equivalents.

Архітектура

Decoder-only Transformer (dense, Grouped Query Attention)

Llama 3.3 70B Instruct was released by Meta on 6 December 2024 as the final 3.x release before Llama 4. It is a dense decoder-only Transformer with 70 billion parameters, 80 layers, 64 query heads, 8 KV heads (Grouped Query Attention) and a tokenizer with 128K vocabulary. The notable claim is that the post-training recipe lifts the 70B model to match or exceed Llama 3.1 405B on most benchmarks at roughly 1/6th of the inference cost. Llama 3.3 reuses the Llama 3.1 pretrained base (which was trained on approximately 15 trillion tokens of curated public web data, code, books and licensed datasets, with a December 2023 knowledge cutoff). The improvement comes from an updated post-training pipeline combining new supervised fine-tuning, rejection sampling, Direct Preference Optimisation (DPO), online reinforcement learning, and synthetic instruction data generated by Llama 3.1 405B and other models. Llama 3.3 supports 128K context, function calling, parallel tool calls, JSON output and the official Llama 3 chat template. The model is text-only (vision sits in the Llama 3.2 family) and ships under the Llama 3.3 Community License which permits commercial use except for products with more than 700 million monthly active users at launch.

Параметри

70B (dense)

Контекст

128 000 токенів

Можливості

• 70B dense parameters with Grouped Query Attention
• Post-training lifts 70B to roughly match Llama 3.1 405B on key benchmarks
• 128K context window
• Function calling and parallel tool calls (Llama 3.1+ style)
• JSON output with the chat template tool format
• Multilingual: 8 officially supported languages including English, German, French, Italian, Portuguese, Hindi, Spanish, Thai
• Code generation across major programming languages
• Open weights under Llama 3.3 Community License
• Massive ecosystem: vLLM, SGLang, TGI, llama.cpp, Ollama, MLX, HuggingFace
• GGUF/AWQ/GPTQ quantised variants in the community
• Best for: open-weight chat, function calling, on-prem enterprise, cost-efficient replacement for 405B.

Навчання та ліцензія

Reuses the Llama 3.1 pretrained base (15T tokens of curated public web data, code, books and licensed data, with a December 2023 cutoff). Post-training applies SFT, rejection sampling, DPO, online RL and synthetic data from Llama 3.1 405B.

Ліцензія: Llama 3.3 Community License: open weights, commercial use permitted, with a >700M MAU clause that requires a separate license from Meta.

Тестування безпеки: Meta applies SFT and RL safety stages plus a 'cybersec' and 'malicious code' evaluation pipeline. The Llama 3 family is among the most safety-tested open-weight model lines, with published Responsible Use Guides and the Llama Guard 3 companion classifier.