Mamba: The Architecture That Set Out to Kill Transformers
Moonshot AI
The Mamba architecture, based on selective state spaces (SSM), was introduced as an alternative to transformers, promising linear complexity instead of quadratic. In practice, however, Mamba did not replace transformers but entered hybrid models, where it is combined with attention layers: roughly one attention layer per seven Mamba layers.
In December 2023, researchers Albert Gu and Tri Dao introduced the Mamba architecture, based on selective state space models (SSMs). Unlike transformers, where each token is compared with all others (quadratic complexity O(N²)), Mamba uses a single fixed-size hidden state that is updated sequentially with linear complexity. The key innovation is a selective mechanism: the state update parameters (A, B, C) are recomputed for each token, allowing the model to decide what information to retain and what to discard, similar to the attention mechanism but without quadratic cost. Thanks to the associativity of computations, Mamba training can be parallelized, solving the slow training problem inherent in old RNNs. However, Mamba has drawbacks: the fixed state compresses information, degrading precise detail extraction and copying ability compared to transformers, which explicitly store all tokens. Therefore, Mamba has not replaced transformers but is used in hybrid models, where roughly one attention layer is added for every seven Mamba layers. This combination preserves Mamba's cheap long memory and transformers' precise retrieval. As a result, many modern models may implicitly use Mamba.
Source: Habr — хаб NLP —
original