/images/avatar.png

Edge AI Deployment: Running LLMs on Device

Edge AI means on-device LLMs keep data local and skip API bills. Q4 3B–7B models already produce tokens on machines like iPhone 15 Pro and MacBook M3. Ollama is the fastest entry; Apple Silicon uses MLX, everything else uses llama.cpp. Fit for offline drafts and privacy-sensitive queries, not as a drop-in cloud-model replacement.

Early 2026 Edge AI is already usable.

iPhone 15 Pro + 3B Q4 model, daily simple tasks sufficient. Mac Studio can run 13B model, most scenarios comparable to cloud API.

Specialized Models for Code, Speech, and Images

Codestral is the current reason to keep a specialized code model in the stack: fill-in-the-middle at IDE latency, not another chat endpoint. General models in the GPT-5.6 family, Claude, and Gemini still own planning, tools, and mixed context. No local bake-off, so no scoreboard. The only numbers below are vendor claims with a live official URL.