Running Local LLMs on iPhone: Memory, Core ML, MLX, and Limitations
Apple
Meta
The article explores how to run local LLMs on iPhone, covering Apple Silicon architecture, unified memory, quantization, KV cache, Core ML vs MLX, thermal throttling, and a hybrid pipeline. It concludes that local models are justified only for privacy-critical or offline scenarios, while cloud models are superior otherwise.
The author, an iOS developer, explains the challenges of running local LLMs on iPhone. Apple Silicon uses unified memory shared by CPU, GPU, and ANE, with limited available memory for apps. Quantization reduces model size (e.g., FP16 to INT4) but may degrade accuracy. KV cache grows with conversation length and can trigger Jetsam (system memory killer). Core ML offers static optimization and AOT compilation, supporting ANE but with context length limitations. MLX uses lazy evaluation and dynamic graphs, performing better in autoregressive loops but causing more heat and throttling. Thermal throttling reduces GPU performance to keep device temperature below 43–45°C for user comfort. The author recommends monitoring thermal state and suggests a hybrid pipeline for stable generation.
- Сокращения
- LLM = Large Language Model — большая языковая модель
- CPU = Central Processing Unit — центральный процессор
- GPU = Graphics Processing Unit — графический процессор
- ANE = Apple Neural Engine — нейронный движок Apple
- RAM = Random Access Memory — оперативная память
- VRAM = Video Random Access Memory — видеопамять
- UI = User Interface — пользовательский интерфейс
- KV = Key-Value — ключ-значение
- AOT = Ahead-of-Time — предварительная компиляция
- PC = Personal Computer — персональный компьютер
- PCIe = Peripheral Component Interconnect Express — шина PCI Express
Source: Habr — хаб ИИ —
original