Rob Hoeijmakers stared at his workbench jar of screws for years. Fishing for the right size always failed. Then ChatGPT entered the picture. He snapped photos, got names, sorted everything by type, thread, length. Order emerged. HTTP caching felt just like that jar—opaque, despite three decades of specs.
Hoeijmakers, a web builder since the early 1990s, finally conquered it. Not through fresh docs. AI tools like Claude did the heavy lifting. They dissected his Cloudflare Workers setup, headers, TTLs, edge versus browser behavior. An afternoon’s back-and-forth yielded a strategy he could explain. Hoeijmakers.net details the win: consistent edge caching for his Ghost-powered blog, global readers in tow.
Caching headers. Cache-Control. ETags. Vary. Pros knew the terms. Application stayed foggy. Every serious try hit a context wall—CDN quirks, invalidation pitfalls, browser oddities. Documentation sat there, dense and unyielding. Implement something plausible. Nod. Move on, doubts lingering.
AI changed that. Claude held the full picture. Questions got tailored answers for his exact stack. Threads unraveled inconsistencies. Revisions flowed. Checks confirmed. What took weeks or specialists compressed into hours. The result? Headers that deliver intent. Edge rules predictable.
Why Now? Machines Gobble Content, Ignore JavaScript
Vanity metrics didn’t drive it. PageSpeed scores? Meh. Traffic shifted. Humans still click. But crawlers dominate: search bots, AI trainers, retrieval agents feeding LLMs. These don’t render. No JavaScript wait. Request hits. Response returns. Done.
For them, caching isn’t nice-to-have. It dictates cost, latency, uptime. Hoeijmakers notes a growing crawler share on his site. Optimize HTML at the edge. Global TTLs. Predictable expiry. Not for Singapore users shaving 200ms. For the next fetch—from a bot, then another, then more.
His public dashboard tracks it: human, AI crawler, SEO bot, unknown. Cloudflare tiered caching atop Ghost. Workers log to D1. Real numbers, not theory.
Industry echoes the urgency. Redis, long a caching staple, powers beyond basics. Sorted sets shine for ordered data—leaderboards, time-series. A mobile studio with 12 million daily users hits sub-0.1ms score updates, 0.2ms top-100 queries on 50 million entries. Memcached couldn’t touch it without race conditions. Tech Insider benchmarks Redis winning four of five tests, 2M+ ops/sec.
Redis 8, GA since 2025, amps it up. Vector sets for AI similarity search. Up to 87% faster commands. Strings, hashes, sets, sorted sets—all benchmarked on bare metal, version after version. No regressions. Redis.io.
Sorted sets marry hash tables (O(1) lookups) and skip lists (O(log N) ranks, ranges). Memory tax: 5-6x a plain list. Worth it for real-time ranks. Twitter’s timeline cache leaned on them. Supercell, Riot Games too. The Excited Engineer.
But pitfalls lurk. SORT on large sets? O(N+M log M)—slow. Use ZRANGE instead; native order. Store sorted results. Refresh periodically. Redis 7’s SORT_RO hits replicas read-only. Oneuptime.
From Personal Fix to Infrastructure Backbone
Hoeijmakers’ afternoon hack scales bigger. Publishers face bot floods. AI pipelines scrape relentlessly. Caching turns expense into efficiency. Edge-global HTML. No re-renders. Machines thrive.
Recent X chatter reinforces. Devs pick Redis sorted sets for leaderboards—1M+ players, real-time ranks. O(log N) tops PostgreSQL sorts. @Ashutosh_Z_. Gaming studios swap MySQL+Memcached for Redis; latency drops 15-20ms. Workers retry via sorted sets, Lua scripts. @Arsalan_0101.
Yet Redis isn’t alone. Valkey forks emerge post-licensing shifts. Dragonflydb eyes richer types. Benchmarks rage on. But for HTTP edges like Cloudflare, AI-clarified headers rule. Bots don’t negotiate.
Three decades unsolved. AI named the screws. Sorted. Machines wait for no one.
WebProNews is an iEntry Publication