Snowflake Arctic Instruct

Σχετικά με Snowflake AI Research

Ιδρύθηκε 2023 · San Mateo, California, USA

Snowflake AI Research is the applied-AI team inside the Snowflake data cloud company, established as a focused unit in 2023 under Yuxiong He (former Microsoft DeepSpeed lead) and Samyam Rajbhandari. Snowflake itself is a public NYSE company (founded 2012) headquartered in Bozeman, Montana with engineering primarily in California. The AI Research team built Arctic to demonstrate that enterprise-targeted LLMs (SQL, code, instruction following) could be trained cheaply by combining many-small-expert MoE designs with carefully curated training data. Arctic Instruct was released April 2024 under Apache 2.0, with the team publicly reporting a training cost of approximately $2M — a fraction of comparable Western frontier runs.

Αρχιτεκτονική

Hybrid Dense-MoE Transformer

Snowflake Arctic Instruct is a hybrid dense-MoE transformer released April 2024 under Apache 2.0. The architecture combines a 10B dense transformer with a 128-way MoE component (128 experts of ~3.66B parameters each, top-2 routing) for a total of 480B parameters and 17B active per forward pass. Compared to peer MoEs (Mixtral 8x22B, DBRX) Arctic uses many more, smaller experts and a high expert-count-to-active-param ratio — a deliberate choice to maximise specialisation for enterprise SQL and code tasks at low active compute. Training used a three-stage curriculum on roughly 3.5T tokens of web, code, GitHub, StackExchange and Snowflake-curated enterprise SQL data, with the final stage heavily oversampling SQL and structured outputs. Snowflake reported total training compute of approximately $2M on a cluster of around 3,200 H100 GPUs. Post-training was supervised fine-tuning plus DPO on enterprise instruction data; no full RLHF pipeline was reported.

Παράμετροι

480B total, 17B active per token (10B dense + 128 x 3.66B experts, top-2 routing)

Περιεχόμενο

4.096 tokens

Δυνατότητες

• Hybrid 10B-dense + 128-expert MoE architecture
• 17B active parameters out of 480B total — cheap inference for an MoE this large
• Strong text-to-SQL performance (matches or beats Llama 3 70B on Spider, BIRD)
• Solid enterprise code generation in Python, Java, SQL
• Instruction following tuned for structured (JSON) outputs
• Apache 2.0 open weights — fully permissive commercial use
• Reported $2M training cost — milestone in cheap-to-train frontier MoE
• Best for: enterprise SQL generation, structured extraction, self-hosted business assistants.

Εκπαίδευση & άδεια

Pretrained on approximately 3.5 trillion tokens in a three-stage curriculum: broad web data, code-and-math-heavy mix, and a final SQL-and-enterprise-instruction-heavy phase. Sources include filtered Common Crawl, GitHub, StackExchange, books, and Snowflake-curated enterprise SQL corpora. Knowledge cutoff is early 2024. Post-training is supervised fine-tuning plus DPO on enterprise instruction data.

Άδεια: Apache 2.0 for both base and Instruct weights. Commercial use, redistribution and modification permitted without royalty.

Δοκιμές ασφάλειας: Internal Snowflake safety review; no formal red-team report published. The model is positioned for enterprise back-end use rather than consumer chat, so guardrails are lighter than chat-tuned frontier models.