AI21 Labs

From GPT-2 to gpt-oss: Analysis of Architectural Improvements and Comparison with Qwen3

OpenAI has released its first open-weight models since 2019 — gpt-oss-120b and gpt-oss-20b. The article examines key architectural changes compared to GPT-2: removal of dropout, replacement of absolute positional embeddings with RoPE, transition from GELU to SwiGLU, introduction of mixture of experts (MoE) and grouped query attention (GQA). It also discusses MXFP4 optimization for running on a single GPU and comparison with Qwen3 and GPT-5.

OpenAI Alibaba/Qwen Meta Google/DeepMind AI21 Labs Tencent