AMD Releases Instella-MoE-16B-A3B: A Fully Open Mixture-of-Experts LLM with 2.8B Active Parameters Trained on Instinct GPUs
AMD introduced Instella-MoE-16B-A3B, a fully open Mixture-of-Experts language model with 16 billion total parameters and 2.8 billion active per token, trained from scratch on Instinct MI300X and MI325X GPUs. The release includes weights, data mixtures, training configs, and inference code, with innovative features like Gated Multi-head Latent Attention and FarSkip-Collective connectivity.
AMD released Instella-MoE-16B-A3B, a fully open Mixture-of-Experts language model trained from scratch on Instinct MI300X and MI325X GPUs. The model has 16 billion total parameters but activates only 2.8 billion per token, using 2 shared experts and 6 routed experts selected from 64 per MoE layer. AMD is publishing weights from every training stage, along with data mixtures, training configs, and inference code. Two key structural choices are Gated Multi-head Latent Attention, which adds a lightweight learned output gate, and FarSkip-Collective, which overlaps expert-parallel communication with computation. AMD reports a 12.7% pre-training speedup and up to a 39.2% reduction in time to first token when serving with expert parallelism. The model was pre-trained on 7.1 trillion tokens from open corpora, with mid-training using Dolma3 Dolmino 100B, and a long-context stage extending the window from 4K to 64K using YaRN. Post-training includes SFT, DPO, and RL in the Miles framework. The base checkpoint averages 76.7, the strongest among fully open models, ahead of Moonlight-16B-A3B (76.2) and others. The weights are under a ResearchRAIL license for academic and research purposes only, while the training codebase is MIT licensed.
- Сокращения
- MoE = Mixture of Experts — смесь экспертов
- GPU = Graphics Processing Unit — графический процессор
- LLM = Large Language Model — большая языковая модель
- MLA = Multi-head Latent Attention — многоголовое латентное внимание
- BF16 = Brain Floating Point 16 — 16-битное число с плавающей точкой
- RL = Reinforcement Learning — обучение с подкреплением
- SFT = Supervised Fine-Tuning — обучение с учителем
- DPO = Direct Preference Optimization — прямая оптимизация предпочтений
Source: MarkTechPost —
original