TII Falcon 180B Chat

Про Technology Innovation Institute (TII)

Засновано 2020 · Abu Dhabi, United Arab Emirates

The Technology Innovation Institute (TII) is the applied-research pillar of Abu Dhabi's Advanced Technology Research Council (ATRC), established in 2020. TII houses dedicated centres for cryptography, autonomy, robotics, biotechnology and AI. The AI Cross-Center Unit released the Falcon 7B and 40B open-weight LLMs in mid-2023, which briefly topped the Hugging Face Open LLM Leaderboard. Falcon 180B, released September 2023, was for several months the largest openly-available LLM and a milestone in non-US frontier model development. The Falcon series — and TII's broader open-source posture — has positioned the UAE as a serious player in sovereign AI alongside the US, EU and China. The team is led by Hakim Hacid and Ebtesam Almazrouei. Subsequent releases include Falcon 2 (2024) and Falcon 3 (late 2024).

Архітектура

Decoder-only Transformer

Falcon 180B is a dense decoder-only transformer with 80 layers, hidden size 14848, 232 attention heads, and a 2,048-token context window. It uses multi-query attention (a single KV head shared across all 232 query heads) — a memory-efficient design that predates grouped-query attention's popularity — together with parallel attention/MLP layers (GPT-J style), RoPE positional embeddings and FlashAttention kernels. The model was pretrained on 3.5 trillion tokens drawn primarily from RefinedWeb (TII's deduplicated and filtered Common Crawl pipeline) plus curated books, code and conversation data. Training used 4,096 NVIDIA A100 40GB GPUs on AWS SageMaker for approximately seven months, consuming around 7 million GPU-hours. The Chat variant is a supervised fine-tune on a mixture of Ultrachat, Platypus and OpenAssistant — no RLHF stage. Released September 2023 under the TII Falcon License (later replaced by TII Falcon LLM License 2.0 with more permissive terms).

Параметри

180B (dense)

Контекст

2,048 токенів

Можливості

• 180B dense parameters — largest openly-available LLM at September 2023 release
• Multi-query attention for memory-efficient inference
• Pretrained on 3.5T tokens of RefinedWeb-filtered data
• Open weights for both base and Chat variants
• Matched LLaMA 2 70B on many English benchmarks at release
• TII Falcon License permits commercial use (with conditions)
• Trained for ~7M GPU-hours on AWS SageMaker
• Best for: research baselines, legacy 2023-era deployments, sovereign AI demonstrations.

Навчання та ліцензія

Pretrained on 3.5 trillion tokens. The mix is primarily RefinedWeb (TII's deduplicated and filtered Common Crawl), plus curated books, code, and multilingual data. Knowledge cutoff is February 2023. The Chat variant is supervised-fine-tuned on Ultrachat, Platypus and OpenAssistant; no RLHF stage was reported.

Ліцензія: TII Falcon License (later TII Falcon LLM License 2.0). Permits commercial use and redistribution with attribution; earlier license required royalties above $1M attributable annual revenue (removed in v2.0). Less straightforward than Apache 2.0 or MIT — review terms before deploying.

Тестування безпеки: TII published a model card discussing limitations; no formal third-party red-team evaluation. The Chat variant inherits standard SFT-only refusal behaviour without RLHF safety tuning.