Qwen3.5/3.6 on 16 GB VRAM: Quants, Fine-Tune, Comparison, and a Bit about Gemma-4
Alibaba/Qwen
Google/DeepMind
Unsloth
A developer tested various quantized and fine-tuned models (Qwen3.5/3.6, Gemma-4) locally on an RTX 5060 Ti 16GB to fix bugs in their game code. Results show that low quants like IQ3 can work, but quality varies unpredictably between providers and model versions. Fine-tunes are mostly poor, with one exception, and the author recommends testing on your own tasks rather than trusting benchmarks.
The author, a non-professional programmer, created games using AI assistance and decided to run models locally. Their setup was a Ryzen 5 5600G with 32GB RAM and an RTX 5060 Ti 16GB. They tested a zoo of models, including Qwen3.5 and Qwen3.6 in various quants (IQ3XS, Q3K_S, IQ3_S, IQ3_M, 4bpw), fine-tunes of Qwen3.5 and Qwen3.6, and Gemma-4 12b in QAT and Q5K_S. The test involved fixing a bug in a game (map clipping) and cleaning up code. Notable results: Qwen3.5 27b in UD-IQ3XXS succeeded, Qwen3.5 27b in Q3K_S succeeded and even optimized code, Qwen3.6 in IQ3_S also succeeded, while Qwen3.6 in UD-IQ3XXS and Q3K_S failed and broke the game. Gemma-4 12b QAT succeeded but Q5K_S failed. Fine-tunes were mostly poor, except 'defiant fable' which was faster and more efficient. The author found that quant quality depends heavily on the provider (Unsloth vs mradermacher) and that i-matrix quants are not always better. The author concludes that testing on your own tasks is essential, and low quants like IQ3 can be viable.
- Abbreviations
- VRAM = Video Random Access Memory — видеопамять
- QAT = Quantization-Aware Training — обучение с учётом квантования
Source: Habr — хаб ИИ —
original