Developer’s llama.cpp Fork Makes Prompt Lookup Drafting Up to 42x Faster

Hayder Tirmazi rebuilt the n-gram caches behind llama.cpp's prompt lookup decoding, cutting drafting time per token by up to 42x and memory use by up to 2.6x. The changes sit in his fork and haven't been merged upstream.
Developer’s llama.cpp Fork Makes Prompt Lookup Drafting Up to 42x Faster
Developer’s llama.cpp Fork Makes Prompt Lookup Drafting Up to 42x Faster

Prompt lookup decoding in llama.cpp just got a lot cheaper to run, at least in one developer’s fork. In a blog post published Sept. 26, Hayder Tirmazi said a set of data-structure fixes makes drafting for prompt lookup “up to 42x faster while using up to 2.6x less memory,” building largely on the work of Daniel Lemire and Martin Ankerl.

Prompt lookup decoding, also called n-gram speculation, is a form of speculative decoding that swaps the usual small draft model for an n-gram model. The engine checks the most recent tokens against cached n-grams and proposes whatever token usually came next. It needs no extra model, and it does best when a model’s output repeats its input, as in code rewrites. llama.cpp keeps three such caches: a context cache for the current tokens, a dynamic cache built from earlier runs, and a static cache built from a text corpus with llama-lookup-create.

Where the speedup came from

Tirmazi’s first change was close to a bug fix. The inner maps inside the n-gram caches were being copied on every drafting step, and reading them by reference instead made drafting 4.5x to 25.6x faster depending on corpus size.

He then replaced the outer std::unordered_map with Ankerl’s unordered_dense segmented map, swapped the inner maps for sorted vectors (64% of the 2-grams in his static cache had only one follower token), and moved the static cache to Lemire’s constmap, an immutable map built on binary fuse filters. The constmap change alone cut static cache load time for the full 541 MB WikiText-103 corpus from 3.76 seconds to 0.23 seconds, and the cache now takes about as much memory as its file, 463 MB for a 467 MB file.

After the post went up, Lemire sent a pull request that checks whether the most frequent follower token can clear llama.cpp’s acceptance thresholds before scoring the rest. Tirmazi said that made drafting up to 4.2x faster again with a static cache, for a combined speedup of up to 140x.

What it doesn’t change

None of the work alters which tokens get drafted, and Tirmazi said acceptance rates stayed essentially identical. His benchmarks measure drafting latency, cache load time and cache memory, not end-to-end generation speed, and they ran on an Apple M4 Pro with a 4096-token context. Developer Anaz S. Aji, who traced the numbers, noted that the per-token drafting cost on the largest corpus fell from 165.48 microseconds to 3.98, and to 1.18 with Lemire’s precheck.

The bigger catch is where the code lives. The optimizations are open pull requests on Tirmazi’s own fork of llama.cpp, and Aji wrote that none had been sent to upstream llama.cpp. Anyone who wants the faster path has to run the fork, and the gains depend on workloads where output echoes the prompt. For a wider look at running models locally, see WebProNews’ report on local LLMs on everyday CPUs.

Subscribe for Updates

AIDeveloper Newsletter

The AIDeveloper Email Newsletter is your essential resource for the latest in AI development. Whether you're building machine learning models or integrating AI solutions, this newsletter keeps you ahead of the curve.

By signing up for our newsletter you agree to receive content related to ientry.com / webpronews.com and our affiliate partners. For additional information refer to our terms of service.

Notice an error?

Help us improve our content by reporting any issues you find.

Get the WebProNews newsletter delivered to your inbox

Get the free daily newsletter read by decision makers

Subscribe
Advertise with Us

Ready to get started?

Get our media kit

Advertise with Us