Prompt lookup decoding in llama.cpp just got a lot cheaper to run, at least in one developer’s fork. In a blog post published Sept. 26, Hayder Tirmazi said a set of data-structure fixes makes drafting for prompt lookup “up to 42x faster while using up to 2.6x less memory,” building largely on the work of Daniel Lemire and Martin Ankerl.
Prompt lookup decoding, also called n-gram speculation, is a form of speculative decoding that swaps the usual small draft model for an n-gram model. The engine checks the most recent tokens against cached n-grams and proposes whatever token usually came next. It needs no extra model, and it does best when a model’s output repeats its input, as in code rewrites. llama.cpp keeps three such caches: a context cache for the current tokens, a dynamic cache built from earlier runs, and a static cache built from a text corpus with llama-lookup-create.
Where the speedup came from
Tirmazi’s first change was close to a bug fix. The inner maps inside the n-gram caches were being copied on every drafting step, and reading them by reference instead made drafting 4.5x to 25.6x faster depending on corpus size.
He then replaced the outer std::unordered_map with Ankerl’s unordered_dense segmented map, swapped the inner maps for sorted vectors (64% of the 2-grams in his static cache had only one follower token), and moved the static cache to Lemire’s constmap, an immutable map built on binary fuse filters. The constmap change alone cut static cache load time for the full 541 MB WikiText-103 corpus from 3.76 seconds to 0.23 seconds, and the cache now takes about as much memory as its file, 463 MB for a 467 MB file.
After the post went up, Lemire sent a pull request that checks whether the most frequent follower token can clear llama.cpp’s acceptance thresholds before scoring the rest. Tirmazi said that made drafting up to 4.2x faster again with a static cache, for a combined speedup of up to 140x.
What it doesn’t change
None of the work alters which tokens get drafted, and Tirmazi said acceptance rates stayed essentially identical. His benchmarks measure drafting latency, cache load time and cache memory, not end-to-end generation speed, and they ran on an Apple M4 Pro with a 4096-token context. Developer Anaz S. Aji, who traced the numbers, noted that the per-token drafting cost on the largest corpus fell from 165.48 microseconds to 3.98, and to 1.18 with Lemire’s precheck.
The bigger catch is where the code lives. The optimizations are open pull requests on Tirmazi’s own fork of llama.cpp, and Aji wrote that none had been sent to upstream llama.cpp. Anyone who wants the faster path has to run the fork, and the gains depend on workloads where output echoes the prompt. For a wider look at running models locally, see WebProNews’ report on local LLMs on everyday CPUs.


WebProNews is an iEntry Publication