Discrete Quantization: The Next Frontier in Compressing Neural Networks
PrismML
The article explores the shift from classical quantization techniques to discrete quantization, where weights are limited to a small set of values (binary, ternary). It highlights the Bonsai 27B model family from PrismML, which uses binary and ternary versions of Qwen 3.6 27B, achieving up to 14x compression while retaining a significant portion of the original model's capabilities. The author tests the ternary version, noting faster inference but uneven performance drops in complex tasks, and discusses the practical implications for running large models on consumer hardware.
The article begins by discussing the evolution of neural network quantization, moving from traditional floating-point formats to integer types like INT8 and INT4, and then to discrete quantization with binary (1-bit) and ternary (1.58-bit) weights. This approach radically reduces model size and enables deployment on consumer hardware. The Bonsai 27B family from PrismML, based on Qwen 3.6 27B, offers binary and ternary versions that occupy 3.8 GB and 7.17 GB respectively, compared to the original 53.8 GB. The company claims up to 90-95% preservation of original capabilities, but the author's tests show about 75% for math and coding, and 60% for complex multi-agent function calling, while still being impressive for a model 10x smaller. The author notes that the ternary model outperforms the original in speed, generating text faster. Additionally, a speculative drafter DSpark is included to boost generation speed by 35% without quality loss. However, the model requires a special fork of llama.cpp due to non-standard quantization formats, which the author demonstrates how to build and run. The author concludes that this represents a significant achievement, offering quality comparable to larger quantized models while fitting in smaller VRAM, making it suitable for local deployment and potentially on mobile devices like the iPhone 17 Pro Max.
- Сокращения
- FP32 = Floating Point 32-bit — 32-битное число с плавающей точкой
- BF16 = Brain Floating Point 16-bit — 16-битное число с плавающей точкой (формат bfloat16)
- FP16 = Floating Point 16-bit — 16-битное число с плавающей точкой
- INT8 = Integer 8-bit — 8-битное целое число
- INT4 = Integer 4-bit — 4-битное целое число
- QAT = Quantization-Aware Training — обучение с учетом квантования
- AWQ = Activation-aware Weight Quantization — квантование весов на основе активаций
- LLM = Large Language Model — большая языковая модель
- GPU = Graphics Processing Unit — графический процессор
- CPU = Central Processing Unit — центральный процессор
- VRAM = Video Random Access Memory — видеопамять
- NVME = Non-Volatile Memory Express — энергонезависимая память (интерфейс NVMe)
- VRM = Voltage Regulator Module — модуль регулятора напряжения
- GGML = GPT-Generated Model Language (format) — формат файлов моделей GGML
Source: Habr — хаб ИИ —
original