AI21 Jamba 1.5 Large

Σχετικά με AI21 Labs

Ιδρύθηκε 2017 · Tel Aviv, Israel

AI21 Labs is one of the earliest commercial LLM companies, founded in 2017 in Tel Aviv by Yoav Shoham (Stanford emeritus, AI pioneer), Ori Goshen and Amnon Shashua (Mobileye founder, ex-Intel SVP). AI21 built the Jurassic-1 (2021) and Jurassic-2 (2023) families and pioneered hybrid State-Space + Transformer architectures with Jamba in March 2024 — the first production-scale Mamba-Transformer hybrid LLM. Jamba 1.5 followed in August 2024 at two scales: Mini (52B total / 12B active) and Large (398B total / 94B active). AI21 has raised over $336M from investors including Google, Nvidia, Walden Catalyst and Pitango, and serves enterprise customers through AI21 Studio, AWS Bedrock, Azure AI Studio, and Snowflake Cortex.

Αρχιτεκτονική

Hybrid Mamba-Transformer Mixture-of-Experts

Jamba 1.5 Large is a hybrid State-Space + Transformer Mixture-of-Experts model. The architecture interleaves Mamba (selective state-space) layers with standard self-attention layers in a 7:1 Mamba-to-Attention ratio across 72 blocks. Mixture-of-Experts is applied to MLP modules in attention blocks with 16 experts and top-2 routing, giving 94B active parameters out of 398B total. The Mamba layers handle long-range dependencies with O(N) memory while the attention layers preserve in-context retrieval quality, enabling a true 256,000-token effective context — empirically validated on the RULER long-context benchmark, where pure-transformer 128K models degrade noticeably. The model uses a 64,000-token BPE tokeniser and supports nine languages (English, Spanish, French, Portuguese, Italian, Dutch, German, Arabic, Hebrew). Released August 2024 under the Jamba Open Model License with hosted access via AI21 Studio, AWS Bedrock, Azure AI Studio and Snowflake Cortex.

Παράμετροι

398B total, 94B active per token (16 experts, top-2 routing)

Περιεχόμενο

256.000 tokens

Δυνατότητες

• Hybrid Mamba+Transformer+MoE architecture
• 398B total / 94B active parameters
• 256K effective context — best-in-class on RULER long-context benchmark
• Constant memory per token from Mamba — cheap long-context inference
• Native function calling and JSON-mode structured output
• Multilingual: English, Spanish, French, Portuguese, Italian, Dutch, German, Arabic, Hebrew
• Open weights on Hugging Face under Jamba Open Model License
• Best for: long-document analysis, many-document RAG, long-trace agents, cost-efficient enterprise long-context inference.

Εκπαίδευση & άδεια

Pretrained on trillions of tokens of web data, code, math, books and multilingual sources (exact figure not disclosed). Knowledge cutoff March 2024. Post-training is supervised fine-tuning plus preference optimisation; the SSAM (state-space attention mix) post-training adapts Mamba-state regularisation.

Άδεια: Jamba Open Model License. Permissive for research and commercial use with attribution and AUP compliance — weaker than Apache 2.0 but more open than research-only licenses. Hosted commercial access via AI21 Studio, AWS Bedrock, Azure AI Studio and Snowflake Cortex.

Δοκιμές ασφάλειας: AI21 publishes a model card with safety evaluations and limitations. Enterprise positioning emphasises grounded outputs and reduced hallucination over consumer-style chat safety tuning.