Observation notebook · Anya · 2026 · benchmarking branch
MLX is the fastest local inference runtime*†
SpecimenMiniCPM5-1B · MLX / llama.cpp / PyTorch-MPS · base M4 Mini, 16 GB
Statusspeed + RAM measured 2026-06-21 · accuracy pending
Sourceunified_bench.py · 3 adapters · 438+ rows

* on a 1B dense model, batched decode only, M4 Mini, custom generation loop, raw tok/s. † conditions definitely apply. Measured uniformly, "MLX ≈ llama.cpp" holds for single-stream decode; the gap only opens in batched decode, and it costs more than double the RAM.

§1No table settles this, and the flag that makes the RAM numbers comparable

If you run local models on a Mac, you've read the common recommendation to use MLX since it's native and faster. No definitive version of the table exists: runtime × quant method × batch size × {prefill, decode} × RAM, all measured in one equal environment on the same model, because the answer shifts enough by model and workload that one general table wouldn't hold up. Pieces of it exist, mostly single-stream, long-context runs (arguably the more realistic workload), but nothing exhaustive focused on peak throughput, which is what a parallelizable sweep harness actually spends. The full grid is missing for a mundane reason: collecting it needs the same model across a dozen quant formats, three runtimes installed and warmed simultaneously, and one harness measuring the same thing across all of them. The base M4 Mini, 16 GB, gets benchmarked plenty (it's a common starter local-LLM machine), but I'm apparently the only idiot trying to run layer sweeps on small models on hardware better suited to browser tabs.

So I built it myself: one driver (unified_bench.py), three backend adapters (MLX in-process, llama.cpp via llama-server, PyTorch/MPS in-process), identical sweep grids and schema, and one RAM metric applied identically to all three. The model is MiniCPM5-1B (a plain 1.08B Llama, 24 layers): small enough to sweep exhaustively, architecturally vanilla enough that the results aren't a quirk of some exotic block.

llama.cpp mmaps its GGUF, and macOS phys_footprint excludes file-backed mapped pages — so a 640 MB model reads as ~0.10 GB of RAM, and any memory comparison against MLX is meaningless without correction. The fix is --no-mmap. Take one thing from this post for your own benchmarking: that flag, or your RAM numbers are fiction. The single methodological detail that silently breaks every naive comparison · §1

§2No overall winner — each runtime owns a different axis

Measure every axis uniformly, and there is no overall winner. Each runtime owned a different axis in these runs:

AxisWinnerNumbers
Batched decodeMLX, ~1.7–2.4×MLX 1400–1670 vs llama 690–750 tok/s aggregate
Prefill (large prompt)llama.cppllama ~2026 @ pp512 vs MLX ~1700 (MLX plateaus ~pp128, dips by pp512)
Single-stream decodetieboth ~100–155 tok/s at 4-bit (2-bit → ~218)
RAM at peak concurrencyllama.cpp ≪ MLXllama 1.2–2.3 GB vs MLX 10–12.7 GB
PyTorch/MPSlast on every speed axis(but mid on RAM)

The headline numbers come from unified_results.jsonl — 438+ rows spanning runtime × quant method × batch-width grid × prefill-sequence sweep. Everything below is the unpacking, and most of it is more useful than the table.

Throughput was quant-agnostic. At peak concurrency all MLX quants landed at ~1400–1670 tok/s (a ~15% spread) and all healthy llama quants at ~690–750. The M4 saturated on compute and bandwidth regardless of how the weights were packed. 2-bit, 4-bit, 8-bit, K-quants, IQ-quants — at batch, they converged. So throughput isn't a meaningful discriminator between quants in this run: pick on quality, RAM fit, and surgery-compatibility, not speed.

§3MLX's decode win is occupancy, not kernels

Why does MLX win batched decode if the quant doesn't matter? Not better arithmetic: single-stream, MLX and llama are tied, same tok/s, same RAM. The gap opens only as you add concurrent streams. MLX keeps filling a bigger shared-KV batch (saturating around 192–256 streams) before it tops out, while in this run llama's continuous batcher plateaued earlier and lighter (64–128 streams; whether that's an architectural limit or a tuning artifact wasn't isolated). It's scheduler and allocator aggressiveness: MLX trades RAM for batch occupancy. That's the whole mechanism, or at least the simplest one consistent with what was observed.

↳ at a single stream all runtimes coincide — the MLX advantage is entirely about packing more concurrent streams

The occupancy story Aggregate decode throughput against concurrent streams, one curve per runtime: MLX climbs to about 1670 tok/s and saturates late, llama plateaus near 750 by 128 streams, and PyTorch stays low. All runtimes coincide at a single stream, so the MLX advantage is entirely about packing more concurrent streams; the faint per-line band is the spread across quants, tight enough to confirm throughput is quant-agnostic. MLX's batched-decode lead is occupancy, not faster kernels Decode tok/s vs concurrent streams · MiniCPM5-1B, M4 Mini · line = 4-bit quant, band = all-quant spread 0 500 1000 1500 1 8 32 64 128 192 256 MLX → 1671 @ 256, saturates late llama.cpp plateaus by 128 (~750) PyTorch capped @32 1 stream: bands overlap, 50–125 tok/s — no gap yet MLX · 4-bit line, 12-quant band llama.cpp · 22-quant band PyTorch fp16 concurrent streams (decode) aggregate decode tok/s Single-stream they tie; MLX's lead is filling a larger shared-KV batch — occupancy, not arithmetic.
The occupancy story Aggregate decode throughput against concurrent streams, one curve per runtime: MLX climbs to about 1670 tok/s and saturates late, llama plateaus near 750 by 128 streams, and PyTorch stays low. All runtimes coincide at a single stream, so the MLX advantage is entirely about packing more concurrent streams; the faint per-line band is the spread across quants, tight enough to confirm throughput is quant-agnostic. MLX's batched-decode lead is occupancy, not faster kernels Decode tok/s vs concurrent streams · MiniCPM5-1B, M4 Mini line = 4-bit quant · band = spread across all quants 050010001500 183264128192256 1671 ~750 MLX · 4-bit line, 12-quant band llama.cpp · 22-quant band PyTorch fp16 (capped @32) Single-stream they tie; MLX's lead is filling a larger shared-KV batch — occupancy, not arithmetic.
Fig 1 — the occupancy story. Aggregate decode tok/s against concurrent streams, one curve per runtime. MLX climbs to ~1670 and saturates late; llama rises then plateaus ~750 by 128 streams; PyTorch stays low (capped at 32). The faint band around each line is the spread across quants — tight enough to confirm throughput is quant-agnostic.
source unified_results.jsonl (464 rows)type fully grounded, measuredsafety runtimes by marker shape + viridis hue, not red/green

§4Throughput per gigabyte is what decides it on 16 GB

MLX buys that decode throughput with ~5–6× the RAM. At peak concurrency MLX sat at 10–12.7 GB (the only runtime that flirts with the M4's 14 GB Metal wired limit), while llama held 1.2–2.3 GB. On a 16 GB box, this, not tok/s, is the deciding factor, and it inverts the leaderboard.

Score by throughput per gigabyte, the figure of merit that matters when memory is the constraint (which on consumer Apple Silicon it always is), and llama.cpp dominates by 4–5×. Best llama (IQ2_XS) hit ~615 tok/s/GB; every llama quant (~490–615) beat every MLX quant (~114–120); PyTorch trailed at ~56. MLX is the raw-speed-at-any-RAM-cost choice; llama.cpp is the efficient choice, and on a 16 GB machine efficiency is what runs.

↳ the two panels rank the same 11 configs; only the metric changes

The leaderboard flips when you divide by RAM Two panels ranking the same eleven configurations, changing only the metric. Top: raw aggregate decode tok/s, where MLX configs win. Bottom: tok/s per GB of peak RAM, where every llama config now beats every MLX config by four to five times, with IQ3_M excluded as EOS-broken. On 16 GB, the speed leaderboard inverts once you divide by RAM MiniCPM5-1B · M4 Mini · peak batched decode · same 11 configs, same order both panels · RAM via --no-mmap MLX llama.cpp PyTorch RANKED BY RAW DECODE · tok/s aggregate (RAM shown) MLX bf16 1669 12.5GB MLX 2-bit 1483 12.6GB MLX 4-bit 1459 10.2GB MLX OptiQ-4b 1445 10.3GB MLX 8-bit 1435 10.8GB llama F16 778 3.8GB llama Q8_0 745 2.0GB llama IQ4_XS 731 2.0GB llama IQ2_XS 726 1.2GB llama Q4_K_M 721 2.0GB PyTorch fp16 368 6.6GB a clean descending ramp — MLX takes the top five RANKED BY THROUGHPUT PER GIGABYTE · tok/s ÷ peak RAM MLX bf16 134 MLX 2-bit 118 MLX 4-bit 143 MLX OptiQ-4b 140 MLX 8-bit 133 llama F16 207 llama Q8_0 372 llama IQ4_XS 367 llama IQ2_XS 615 llama Q4_K_M 353 PyTorch fp16 56 same order, ramp shattered — the top five (MLX) are now the shortest above PyTorch Every llama.cpp quant beats every MLX config 2–4× per GB. On 16 GB, per-GB is the axis that fits. MLX RAM is a cumulative high-watermark (overstated); llama is per-process fresh — gap direction is conservative. IQ3_M excluded (EOS-broken in this run).
The leaderboard flips when you divide by RAM Two panels ranking the same eleven configurations, changing only the metric. Top: raw aggregate decode tok/s, where MLX configs win. Bottom: tok/s per GB of peak RAM, where every llama config now beats every MLX config by four to five times, with IQ3_M excluded as EOS-broken. On 16 GB, the speed leaderboard inverts once you divide by RAM MiniCPM5-1B · M4 Mini · peak batched decode same 11 configs, same order both panels · --no-mmap MLX llama.cpp PyTorch RAW DECODE · tok/s (peak batch) MLX bf161669 MLX 2-bit1483 MLX 4-bit1459 MLX OptiQ-4b1445 MLX 8-bit1435 llama F16778 llama Q8_0745 llama IQ4_XS731 llama IQ2_XS726 llama Q4_K_M721 PyTorch fp16368 clean ramp — MLX takes the top five PER GIGABYTE · tok/s ÷ peak RAM MLX bf16134 MLX 2-bit118 MLX 4-bit143 MLX OptiQ-4b140 MLX 8-bit133 llama F16207 llama Q8_0372 llama IQ4_XS367 llama IQ2_XS615 llama Q4_K_M353 PyTorch fp1656 ramp shattered — MLX now shortest above PyTorch Every llama.cpp quant beats every MLX config 2–4× per GB — the axis that fits on 16 GB. MLX RAM is a cumulative high-watermark (overstated); llama is per-process fresh. IQ3_M excluded (EOS-broken).
Fig 2 — the leaderboard flips when you divide by RAM. Same configs, same order, two panels. Top: raw aggregate decode tok/s (MLX configs win). Bottom: tok/s per GB of peak RAM (every llama config now beats every MLX config by 4–5×). IQ3_M excluded — EOS-broken in this run. Runtimes grouped by hatching, not colour.
source unified_results.jsonl + m5_air_vs_m4_mini_minicpm5.mdtype fully groundedexcluded IQ3_M (EOS-broken)

§5PyTorch is last; prefill is compute-bound; the arithmetic floor

The fp16 fairness triad (identical weights, all three runtimes): MLX 1669 tok/s @ 256 streams / 12.45 GB; llama 778 @ 128 / 3.76 GB; PyTorch 368 @ 32 / 6.63 GB. PyTorch is ~4.5× behind MLX batched and ~3× behind llama, from eager/unfused-kernel overhead that persists even at matched precision. "Don't serve inference through HuggingFace on a Mac" appears to be correct, and now it has numbers, at least for this model and batch configuration. (Note: the IQ3_M llama row is EOS-broken here, total_tokens=1: its 182 tok/s reading is garbage and is excluded throughout.)

Prefill was even more saturated than decode — ~2000 tok/s at pp512 with only ~12% spread across the whole quant ladder. Q4_0 tiles cleanest into Metal's GEMM and led; K-quants pay a block-structure tax and roll off after pp256; IQ types flatten or keep climbing. None of it was large enough to change a practical decision.

Why quant choice doesn't move the needle: the workload is pinned to a physical limit. Single-stream decode is memory-bandwidth-bound: a 4-bit 1.08B model is ~0.6 GB, so 120 GB/s ÷ 0.6 ≈ 200 tok/s; observed best ~155 = ~78% of that bound. 2-bit (~0.35 GB) predicts ~343, observed 218 ≈ 64%. Prefill is compute-bound: 2000 tok/s × ~2·1.08e9 FLOP/tok ≈ 4.3 TFLOP/s, in the M4 10-core GPU's fp16 range. Which is exactly the kind of answer no general table would have given me: for speed, quant choice wasn't a meaningful discriminator on this model and workload, pick the quant on quality.

§6Addendum — M5 Air vs M4 Mini, and the roofline

A cross-machine run (same model, same MLX/mlx-lm stack, the lean decode loop) on a borrowed fanless M5 MacBook Air against the M4 Mini: the M5 was ~2.3× across the board, decode (3700 vs 1615 peak tok/s), prefill (5350 vs 2200), wall (2700 vs 1162). That's far above the 1.27× memory-bandwidth ratio (153 vs 120 GB/s).

The leading hypothesis: the win is GPU compute and dispatch, not bandwidth. The ~200 tok/s ceiling is single-stream (bandwidth-bound); at batch width 64–384 decode appears compute-bound (weights reused across the batch, past the roofline knee), so it tracks prefill at ~2.3×, both riding the M5's ~2× GPU compute. No decode-specific magic; just which side of the roofline both kernels are on.

↳ the roofline argument made visible — bandwidth-limited at width 1, compute-limited as the batch grows

The roofline crossover The measured M5/M4 decode-throughput ratio across batch widths 64 to 384, rising from 1.44 toward about 2.3, with reference lines at the 1.27× bandwidth ratio and the 2.3× compute ratio. Sub-64 widths are hatched as unmeasured with no curve drawn through them — the crossover below width 64 is a stated, unrun prediction, bandwidth-limited when narrow and compute-limited as the batch grows. The M5/M4 speedup is GPU compute, not bandwidth MLX decode ratio (M5 Air ÷ M4 Mini) vs batch width · MiniCPM5-1B · matched script + reboot on both machines 1.01.52.02.5 6496128192256384 <64: unmeasured (harness floors at 64) 1.27× — memory-bandwidth ratio (153 vs 120 GB/s) ~2.3× — observed plateau (GPU-compute-bound) 1.442.152.131.962.242.29 w64 sits low — nearest the bandwidth bound ratio rides the compute ceiling once batch ≥ 96 batch width (concurrent streams) M5 / M4 decode ratio M5 Air = fanless burst, on battery (sustained TBD); M4 Mini sustained — direction holds, magnitude is a burst upper bound.
The roofline crossover The measured M5/M4 decode-throughput ratio across batch widths 64 to 384, rising from 1.44 toward about 2.3, with reference lines at the 1.27× bandwidth ratio and the 2.3× compute ratio. Sub-64 widths are hatched as unmeasured with no curve drawn through them — the crossover below width 64 is a stated, unrun prediction, bandwidth-limited when narrow and compute-limited as the batch grows. The M5/M4 speedup is GPU compute, not bandwidth MLX decode ratio (M5 Air ÷ M4 Mini) vs batch width MiniCPM5-1B · matched script + reboot, both machines 1.01.52.02.5 6496128192256384 <64 unmeasured 1.27× bandwidth ratio ~2.3× observed plateau (compute-bound) 1.442.151.962.29 M5 Air = fanless burst, on battery (sustained TBD); M4 sustained. Magnitude is a burst upper bound.
Fig 3 — the roofline crossover. Measured M5/M4 decode-throughput ratio across batch widths 64–384 (1.44 → ~2.3), with reference lines at 1.27× (bandwidth ratio) and 2.3× (GPU-compute ratio). Sub-64 widths are hatched as unmeasured — no speculative curve is drawn through them; the crossover below width 64 is the stated, unrun prediction.
source M5/M4 cross-machine run, measured w64–384type measured band + stated prediction (sub-64 unrun)safety reference lines dashed/labelled, not colour-coded

Caveats

This benchmark's scope only covers speed and memory usage, and does not address the real accuracy differences between quants. Accuracy is the axis that discriminates the quants for functional usage, but this sets aside that aspect in order to assess runtime factors since often the quant is dependent on maximizing bit weight within hardware limitations.

Build notes & method

Harness, raw data, and viz: unified_bench.py (three adapters, prefill + concurrency sweeps), unified_results.jsonl, make_unified_viz.py. The model is the same MiniCPM5-1B that the layer-surgery work dissects. The equal-environment discipline is the same one behind the summarization benchmark: measure all of them the same way or the comparison means nothing. Built and run locally on a 16 GB Mac Mini.