From GPT-2 to gpt-oss: Analysis of Architectural Improvements and Comparison with Qwen3

OpenAI has finally released its first open-weight models in six years — gpt-oss-120b and gpt-oss-20b. Unlike GPT-2 (2019), the architecture has undergone numerous changes. First, dropout has been completely removed: in LLMs trained for just one epoch on gigantic datasets, it only degrades performance. Instead of absolute positional embeddings as in GPT-2, RoPE (Rotary Position Embedding) is now used, encoding position by rotating query and key vectors. The GELU activation function has been replaced with the computationally cheaper Swish, and the feed-forward block has been turned into a gated linear unit SwiGLU, which offers more expressiveness with fewer parameters. Additionally, the single feed-forward module has been replaced with a mixture of experts (MoE): for the 20B model, only 1 out of 8 experts is activated per token, allowing increased overall model capacity without rising computational costs. Finally, multi-head attention has given way to grouped query attention (GQA) — keys and values are now shared among multiple heads, saving memory and accelerating inference. Thanks to MXFP4 optimization, the 20B model fits on a consumer GPU with 16 GB of memory, and the 120B model fits on a single H100. The article also compares gpt-oss with Qwen3 (demonstrating a balance of width and depth) and touches on features of the new GPT-5.