Nous Hermes 3 405B

Про Nous Research

Засновано 2023 · San Francisco, USA (distributed)

Nous Research is a community-driven open-source AI collective founded in 2023, co-led by Karan 'Teknium' Malhotra, Jeffrey Quesnelle, Bowen Peng and others, with a distributed contributor base of independent researchers. Nous focuses on uncensored, steerable, character-rich fine-tunes of open base models — its Hermes line (Hermes, OpenHermes, Hermes 2, Hermes 2.5, Hermes 3) is the flagship instruction-following family, with Yarn (context-length extension) and Capybara as influential adjacent projects. Hermes 3 405B was released August 2024 in partnership with Lambda Labs (compute) and was the first full-parameter fine-tune of Meta's Llama 3.1 405B base model. Nous raised seed funding from Distributed Global and a16z-affiliated angels in 2024.

Архітектура

Decoder-only Transformer (Llama 3.1 architecture)

Hermes 3 405B is a full-parameter supervised fine-tune (not LoRA) of Meta's Llama 3.1 405B base model. The architecture is unchanged from Llama 3.1: 126 layers, 16,384 hidden size, 128-head grouped-query attention with 8 KV heads, RoPE positional embeddings with the Llama 3 scaling that supports 128K context, SwiGLU activations and the 128,000-token Llama 3 BPE tokeniser. The fine-tune was carried out by Nous Research on approximately 256 NVIDIA H100 GPUs supplied by Lambda Labs, using a curated dataset of around 390M instruction tokens (~2.5M examples) covering role-play, function calling, code, math, RAG, agentic tool use and uncensored creative writing — much of it Nous-curated synthetic from larger models. The model uses a ChatML-style format with native `<tool_call>` JSON-schema tags and `<scratchpad>` chain-of-thought tags. Released August 2024 under the Llama 3.1 Community License.

Параметри

405B (dense)

Контекст

128,000 токенів

Можливості

• Full-parameter fine-tune of Llama 3.1 405B (not LoRA)
• Strong system-prompt steering for persona and rule-set instructions
• Native ChatML `<tool_call>` JSON-schema tags and `<scratchpad>` reasoning tags
• 128K context inherited from Llama 3.1
• Reduced RLHF-style refusals — friendlier for research and creative writing
• Competitive benchmark scores with Llama 3.1 405B Instruct (MMLU, GPQA, math)
• Open weights under Llama 3.1 Community License
• Best for: customisable agents, role-play platforms, function-calling assistants, self-hosted Llama 3.1 alternatives.

Навчання та ліцензія

Supervised fine-tuning on ~390M instruction tokens across ~2.5M examples covering role-play, function calling, code, math, RAG, agent traces and creative writing. Large fraction is Nous-curated synthetic data distilled from larger models. The full Hermes 3 dataset card is published alongside the model. No RLHF / no DPO in the 405B variant. Base model knowledge cutoff December 2023.

Ліцензія: Llama 3.1 Community License. Commercial use permitted, but services with >700M monthly active users require a separate Meta license. Meta's Acceptable Use Policy applies to all derivatives.

Тестування безпеки: Nous explicitly trades RLHF-style safety guardrails for steerability and creative latitude. Operators deploying for consumer products are expected to add their own safety layer. No formal third-party red-team report.