Everyone obsesses over parameter count. Bigger model, better answers — until the tokens crawl out at three per second and the fan on your GPU sounds like a leaf blower. Half the speed gains people chase by moving from a 7B to a 70B model would come from actually tuning the runtime they already have.
Local LLM inference in 2026 is a stack of decisions: quantization scheme, KV cache dtype, batch size, prompt cache, speculative decoding, offload strategy. We tested eight desktop apps for local LLM inference optimization, from the low-level runtimes that expose every knob to the wrapped-up tools that pick sane defaults for you. Pick the one that matches how deep you want to go.
What to look for in a local LLM inference optimization app
Not every runtime is optimising the same thing. Before picking, know which of these you actually need:
- Quantization support. The runtime should read GGUF, EXL2, AWQ, or MLC formats and let you pick the bit width per model. 4-bit is the modern default; some workloads are fine at 3-bit.
- KV cache controls. Cache dtype (fp16 vs q8_0 vs q4_0) and cache size both change VRAM usage substantially. The runtime should let you set them.
- Prompt cache reuse. For chat and RAG, reusing the KV cache from the previous message is the difference between instant follow-ups and a full re-prefill.
- Speculative decoding. Pairing a small draft model with a big target model can double throughput on the same hardware for chat workloads.
- Multi-GPU and offload. Layer-by-layer offload to CPU RAM, tensor split across GPUs, and NUMA-aware allocation matter once models get big.
- Batch and concurrent request handling. If you serve more than one client, throughput at batch size 8 or 16 matters more than latency at batch 1.
Quick comparison
| App | Best for | Platforms | Free plan | Standout feature |
|---|---|---|---|---|
| llama.cpp | Deep quantization + KV cache control | Windows, macOS, Linux | Fully free, open source | GGUF ecosystem, per-layer offload |
| Ollama | Zero-config runtime with sane defaults | Windows, macOS, Linux | Free | One-line model swaps |
| LM Studio | GUI for tuning without a terminal | Windows, macOS, Linux | Free | Visual runtime settings for GGUF |
| vLLM | Batched serving with PagedAttention | Linux, Windows via WSL | Free, open source | Continuous batching for many clients |
| MLC LLM | Compiled inference across GPUs | Windows, macOS, Linux | Free, open source | TVM-compiled kernels, Metal + Vulkan |
| KoboldCpp | Roleplay + creative writing tuning | Windows, macOS, Linux | Free, open source | Per-prompt KV cache tuning UI |
| ExLlamaV2 | Fast EXL2 inference on Nvidia | Windows, Linux | Free, open source | EXL2 quant + speculative decoding |
| TabbyAPI | OpenAI-compatible ExLlamaV2 server | Windows, Linux | Free, open source | Drops behind any OpenAI client |
The apps
1. llama.cpp — Best for deep quantization and KV cache control
llama.cpp is the runtime most of the ecosystem is built on. It reads GGUF quantizations from 8-bit down to 2-bit, offloads per-layer to CPU when the model does not fit in VRAM, and exposes cache dtype settings so you can trade quality for headroom.
The tuning surface is where the speed lives. Setting --n-gpu-layers correctly for your GPU, matching cache dtype to the quant, enabling --flash-attn, and using prompt cache files can double throughput on the same hardware.
Where it falls short: The CLI is dense and the flag list changes fast. First-time tuning takes an afternoon of reading the docs.
Pricing:
- Free, open source.
Platforms: Windows, macOS, Linux — CPU, CUDA, Metal, Vulkan, ROCm.
Download: llama.cpp on GitHub
Bottom line: The runtime to reach for when speed matters more than convenience.
2. Ollama — Best for a zero-config runtime with sane defaults
Ollama wraps llama.cpp and picks defaults that get most people to acceptable throughput without touching a flag. Pull a model, run it, done. The Modelfile format lets you set system prompts, sampling parameters, and context length per model.
For a workstation that runs a handful of models on rotation, this is the fastest path from “installed” to “usable.” Advanced tuning happens through Modelfile params rather than command-line flags.
Where it falls short: The abstraction hides some of the levers that squeeze out the last 30% of throughput. Power users often end up with both Ollama for casual use and llama.cpp for heavy work.
Pricing:
- Free.
Platforms: Windows, macOS, Linux.
Download: Ollama for desktop
Bottom line: The default pick for someone new to local LLMs who wants speed without a manual.
3. LM Studio — Best for tuning without a terminal
LM Studio exposes llama.cpp’s optimisation knobs through a real GUI. Quant selection, n_gpu_layers, context length, cache dtype, and batch size all get sliders and dropdowns instead of flags.
That makes it the best pick for people who understand what the knobs do but do not want to memorise CLI syntax. The built-in chat and API server work as a testing harness while you tune.
Where it falls short: GUI-only means scripting is limited. For headless servers, drop back to llama.cpp or Ollama.
Pricing:
- Free for personal and hobby use.
- Paid: Team plans for commercial deployments.
Platforms: Windows, macOS, Linux.
Download: LM Studio for desktop
Bottom line: The tuning GUI that finally makes llama.cpp accessible.
4. vLLM — Best for batched serving with PagedAttention
vLLM shines when you serve multiple concurrent requests. PagedAttention manages the KV cache like a memory allocator, packing many active sequences into the same VRAM without fragmentation. Continuous batching means new requests slot in between token generations rather than waiting for the current one to finish.
For a home lab that runs an OpenAI-compatible endpoint for several apps at once, vLLM’s throughput curve at batch 8 crushes single-request runtimes.
Where it falls short: Nvidia-first, less mature on AMD and Metal. The tradeoff is you probably want a real GPU for it anyway.
Pricing:
- Free, open source.
Platforms: Linux natively, Windows via WSL.
Download: vLLM on GitHub
Bottom line: The runtime for anyone serving more than one client from the same box.
5. MLC LLM — Best for compiled inference across GPUs
MLC LLM takes a different bet: compile the model kernels with TVM for the exact target hardware. That gives it strong Metal, Vulkan, and WebGPU support, which matters if your desktop is a Mac Studio or an AMD box where CUDA-first runtimes struggle.
The result is fast prefill and decode on hardware that llama.cpp is fine on but not fast on. Compilation adds a one-time step per model per target.
Where it falls short: The model zoo is smaller than GGUF’s, and the workflow is heavier for first-time users.
Pricing:
- Free, open source.
Platforms: Windows, macOS, Linux, plus WebGPU in browsers.
Download: MLC LLM on GitHub
Bottom line: The strongest pick if your primary GPU is not an Nvidia card.
6. KoboldCpp — Best for roleplay and creative writing tuning
KoboldCpp is llama.cpp underneath with a UI aimed at long-form writing and roleplay. That workload leans hard on the KV cache — long context, repeated re-reads of the same story so far — so KoboldCpp exposes cache reuse and context-shift controls prominently.
The scenario tools, world info, and memory features make it a strong pick outside its target audience too. Anyone who runs long-context prompts benefits from the cache tuning.
Where it falls short: UI is functional rather than polished. Server-mode is fine but the primary experience is the built-in chat.
Pricing:
- Free, open source.
Platforms: Windows, macOS, Linux.
Download: KoboldCpp releases on GitHub
Bottom line: The best-tuned front end for long-context work.
7. ExLlamaV2 — Best for fast EXL2 inference on Nvidia
ExLlamaV2 is a CUDA-first inference engine that reads the EXL2 quantization format. On the same Nvidia GPU, it typically beats llama.cpp on tokens-per-second for the same effective quality bit budget. Speculative decoding with a small draft model gives it another jump.
For a workstation with a 3090, 4090, or 5090 as the only accelerator, this is where the speed lives. Pair it with TabbyAPI to get an OpenAI-compatible endpoint.
Where it falls short: Nvidia-only. No Metal, no ROCm story.
Pricing:
- Free, open source.
Platforms: Windows, Linux with Nvidia GPUs.
Download: ExLlamaV2 on GitHub
Bottom line: The runtime for pushing the most tokens per second out of an Nvidia GPU.
8. TabbyAPI — Best for an OpenAI-compatible ExLlamaV2 server
TabbyAPI wraps ExLlamaV2 in an OpenAI-compatible REST API, so anything that already speaks to OpenAI — Cursor, Continue, Aider, LibreChat — can drop it in with a base URL change. Streaming, function calling, and speculative decoding all work through the same API surface.
For someone whose stack already assumes an OpenAI endpoint, TabbyAPI is the shortest path from “local model on disk” to “everything talks to it.”
Where it falls short: Config is TOML files, not a UI. Debugging the first setup takes patience.
Pricing:
- Free, open source.
Platforms: Windows, Linux with Nvidia GPUs.
Download: TabbyAPI on GitHub
Bottom line: The bridge that lets your existing tools use ExLlamaV2 without changing anything else.
How to pick the right one
Pick llama.cpp when you want maximum control and are willing to learn the flags. Everything on this list either uses it or is measured against it.
Pick Ollama when you want speed without a config file. It gets you 80% of the way there in one command.
Pick LM Studio when the flags of llama.cpp look like Chinese and you want the same tuning surface with sliders.
Pick vLLM when the workload is many concurrent requests. Batching wins at scale.
Pick MLC LLM when the primary GPU is Apple Silicon or AMD. CUDA-first runtimes underperform on that hardware.
Pick KoboldCpp when the prompts are long and the story matters. Context handling is its whole game.
Pick ExLlamaV2 (with TabbyAPI) when the box is Nvidia and every millisecond counts.
Stay on the cloud only if the models you need do not have local weights, or the request volume is large enough that a rented A100 is cheaper than the electricity to run a local one.
FAQ
Does quantization actually make local LLMs faster?
Yes, and by more than most people expect. Moving from fp16 to Q5_K_M on the same model typically cuts VRAM use in half and improves tokens per second on GPU-bound loads, with quality loss small enough that most users cannot spot it in blind tests.
What is speculative decoding?
Speculative decoding runs a small “draft” model to guess several tokens ahead, then verifies them in a single big-model pass. When the draft is right, you get multiple tokens per big-model call. On chat workloads, it can nearly double throughput.
What is PagedAttention and why does it matter for local LLMs?
PagedAttention is vLLM’s KV cache management technique. Instead of allocating fixed-size cache slots per request, it treats cache memory as pages that can be assigned dynamically. On a busy server, that means many more concurrent requests fit in the same VRAM.
Which local LLM runtime is fastest on Apple Silicon?
MLC LLM’s compiled Metal kernels are typically the fastest single-user option on Macs. llama.cpp’s Metal backend is close and easier to run — pick MLC if the extra 10 to 20% matters, llama.cpp otherwise.
Can I run a local LLM without a discrete GPU?
Yes. llama.cpp and Ollama both run on CPU with any GGUF model. Speeds on modern laptops range from a few tokens per second on 7B models to a slow crawl on 70B — usable for chat, painful for long generations.