Beyond Standard LLMs: Linear Attention Hybrids, Text Diffusion Models, Code World Models, and Small Recurrent Transformers

The author, Sebastian Raschka, analyzes alternatives to standard autoregressive large language models (LLMs) built on the transformer architecture with attention mechanisms. Among the approaches discussed are linear attention hybrids, which have recently seen a resurgence. These include MiniMax-M1 (456B parameters, mixture of experts (MoE), with lightning attention), Qwen3-Next (which uses Gated DeltaNet and Gated Attention in a 3:1 ratio), DeepSeek V3.2 (sparse attention, sub-linear complexity), and Kimi Linear (also featuring Gated DeltaNet). However, MiniMax-M2 reverted to standard attention due to accuracy issues in reasoning and multitasking tasks. Also mentioned are text diffusion models, code-based world models (for example, for generating game state), and small recurrent transformers (such as a 1.5B model compared to GPT-4 on syntactic parsing tasks).