Beyond Standard LLMs: Linear Attention Hybrids, Text Diffusion, Code World Models, and Small Recursive Transformers
Moonshot AI
MiniMax
Alibaba/Qwen
DeepSeek
Google/DeepMind
Mistral
Meta
Hugging Face
Allen Institute for AI
xAI
OpenAI
IBM
NVIDIA
The article by Sebastian Raschka explores alternatives to standard autoregressive transformer LLMs, including linear attention hybrids (e.g., MiniMax-M1, Qwen3-Next, DeepSeek V3.2, Kimi Linear), text diffusion models, code world models, and small recursive transformers. It discusses the revival of linear attention mechanisms and challenges, such as MiniMax reverting to standard attention in its M2 model due to poor performance on reasoning tasks.
Sebastian Raschka discusses alternatives to standard autoregressive decoder-style transformers, which are built on multi-head attention. Notable models include MiniMax-M1 with lightning attention, Qwen3-Next with Gated DeltaNet and Gated Attention, DeepSeek V3.2 with sparse attention, and Kimi Linear. However, MiniMax reverted to regular attention in its M2 model, citing poor accuracy in reasoning and multi-turn tasks. Qwen3-Next and Kimi Linear use a 3:1 ratio of linear attention (Gated DeltaNet) to full attention (Gated Attention). Gated DeltaNet combines Mamba2 with a delta rule for recurrent state updates. The article also mentions other alternatives like text diffusion models, code world models, and small recursive transformers, but does not detail them.
- Сокращения
- LLM = Large Language Model — большая языковая модель
- MoE = Mixture of Experts — смесь экспертов
- SSM = State Space Model — модель пространства состояний
Source: Sebastian Raschka —
original