Cohere Aya 23 35B

Про Cohere For AI

Засновано 2022 · Toronto, Canada

Cohere For AI (C4AI) is the non-profit research lab of Cohere, founded in 2022 and led by Sara Hooker. Its remit is open science and global multilingual NLP. C4AI's flagship Aya initiative — launched in 2023 — coordinated 3,000+ collaborators across 119 countries to build the largest open-access multilingual instruction-tuning dataset (the Aya Collection: 513M instances across 114 languages). Aya 23, released May 2024, is the second generation of the Aya model family. It builds on Cohere's Command R base architecture and trades broad coverage (101 languages in Aya-101) for stronger per-language quality across 23 widely-used languages. Cohere itself was founded in 2019 by Aidan Gomez (co-author of 'Attention Is All You Need'), Ivan Zhang and Nick Frosst.

Архітектура

Decoder-only Transformer (Command R base architecture)

Aya 23 35B is a dense decoder-only transformer fine-tuned from Cohere's Command R 35B base. The architecture uses grouped-query attention (8 KV heads), RoPE positional embeddings, SwiGLU activations, RMSNorm, no biases, and a 256,000-token multilingual BPE tokeniser that is particularly efficient on non-Latin scripts (Arabic, Hebrew, Korean, Japanese, Chinese). The model was instruction-tuned on the Aya Collection — 513M templated and translated instruction instances — narrowed to coverage for 23 target languages, supplemented by the Aya Dataset (204K human-curated prompt-completion pairs from 65 languages collected via the Aya Annotation Platform) and ShareGPT, Dolly and Flan corpora. There is no public RLHF or DPO stage for Aya 23. Released May 2024 under CC-BY-NC 4.0 alongside an 8B sibling.

Параметри

35B (dense)

Контекст

8 192 токенів

Можливості

• State-of-the-art open multilingual performance across 23 languages
• Particularly strong on Arabic, Hebrew, Persian, Korean, Hindi, Vietnamese
• Outperforms Aya-101 13B, Mistral 7B Instruct v0.2, Gemma 1.1 7B on multilingual benchmarks
• 256K multilingual BPE tokeniser efficient for non-Latin scripts
• Open weights on Hugging Face under CC-BY-NC 4.0
• 8K context window
• Strong instruction following in covered languages
• Best for: research and academic multilingual NLP, non-commercial multilingual prototypes.

Навчання та ліцензія

Instruction-tuned on the Aya Collection (513M instances across 114 languages, filtered to 23 target languages for the 23 variants) plus the Aya Dataset (204K human-curated multilingual prompt-completion pairs from 65 languages collected through the Aya Annotation Platform), ShareGPT, Dolly and Flan. No public RLHF stage. Base model knowledge cutoff March 2024.

Ліцензія: Open weights under CC-BY-NC 4.0 — non-commercial research use only. Commercial use requires the hosted Cohere API under separate commercial terms.

Тестування безпеки: C4AI ran multilingual safety evaluations as part of the Aya 23 release, with a public model card discussing limitations and bias. Refusal behaviour is lighter than Cohere's commercial models.