Nous Hermes 3 70B

Про Nous Research

Засновано 2023 · San Francisco, USA (distributed)

Nous Research is a community-driven open-source AI collective founded in 2023, co-led by Karan 'Teknium' Malhotra, Jeffrey Quesnelle and Bowen Peng with a distributed contributor base. Nous focuses on uncensored, steerable, character-rich fine-tunes of open base models. The Hermes line (Hermes, OpenHermes, Hermes 2, Hermes 2.5, Hermes 3) is its flagship instruction-following family. Hermes 3 70B is the mid-sized variant of the August 2024 Hermes 3 release, fine-tuned from Llama 3.1 70B for steerable behaviour with native tool-use and scratchpad tags — the sweet spot for organisations that can run a 70B model on a small GPU cluster.

Архітектура

Decoder-only Transformer (Llama 3.1 architecture)

Hermes 3 70B is a full-parameter supervised fine-tune of Meta's Llama 3.1 70B base. The architecture is identical to Llama 3.1 70B: 80 layers, 8,192 hidden size, 64-head grouped-query attention with 8 KV heads, RoPE positional embeddings with Llama 3 scaling (128K context), SwiGLU activations, and the 128,000-token Llama 3 BPE tokeniser. The fine-tune mixes role-play, function calling, code, math, RAG, agent traces and creative writing — largely Nous-curated synthetic data distilled from larger models. The model uses ChatML formatting with native `<tool_call>` JSON-schema tags and `<scratchpad>` chain-of-thought tags. A light DPO preference-optimisation stage was applied (unlike the 405B which is SFT-only). Released August 2024 under the Llama 3.1 Community License.

Параметри

70B (dense)

Контекст

128 000 токенів

Можливості

• Full-parameter fine-tune of Llama 3.1 70B with light DPO
• Excellent system-prompt steerability for agents and characters
• Native ChatML `<tool_call>` and `<scratchpad>` tags
• More tractable to self-host than the 405B (~140GB FP16, ~35GB INT4)
• Strong instruction following and creative writing
• 128K context inherited from Llama 3.1
• Comparable to Llama 3.1 70B Instruct on benchmarks with friendlier tool-use format
• Best for: self-hosted assistants, tool-using agents, creative platforms, fine-tuning base for community projects.

Навчання та ліцензія

Supervised fine-tuning on ~390M instruction tokens across ~2.5M examples covering role-play, function calling, code, math, RAG, agent traces and creative writing — largely Nous-curated synthetic data. A light DPO preference-optimisation pass was applied. Base knowledge cutoff December 2023.

Ліцензія: Llama 3.1 Community License. Commercial use permitted, but services with >700M monthly active users require a separate Meta license. Meta's Acceptable Use Policy applies to all derivatives.

Тестування безпеки: Nous explicitly trades RLHF-style safety guardrails for steerability and creative latitude. Operators deploying for consumer products should add their own safety layer. No formal third-party red-team report.