Update, Oct. 26: SQLite’s own write-up now notes that Phil Eaton built a reproducer for the bug that doesn’t depend on SQLite’s special test hooks. His analysis is here.
Tailscale has published the story of a database corruption problem in its control plane that took months to pin down. The cause turned out to be a data race inside SQLite itself, one the SQLite developers believe had been there since version 3.7.0 in 2010.
Tailscale runs SQLite in WAL (write-ahead log) mode and takes manual control of checkpointing, the step that copies new entries from the log into the main database file, so it can run fast, consistent backups. In most deployments SQLite decides when to checkpoint on its own, as The Register explained, and nobody sees it happen.
The breakthrough came after the SQLite developers added a logging shim, tmstmpvfs, to trace checkpoint activity. The next corruption incident showed what was going on. If a write lands at just the wrong moment during a checkpoint, the checkpoint thinks some pages have already been copied into the database file when they haven’t. Those pages are lost, while pages that point to them, such as index entries, still get written. The result is a corrupt file.
What SQLite says
SQLite’s WAL documentation now carries a section on what it calls the WAL-Reset bug. Developer Dan found and fixed it on March 3. It affects versions 3.7.0 through 3.51.2 and is fixed in 3.51.3, released March 13, with backports in 3.44.6 and 3.50.7. It only shows up in WAL mode with two or more connections to the same file trying to write or checkpoint at the same instant.
SQLite calls it serious but not an emergency. By its telemetry, the bug occurs in the wild at about the rate of SSD failures or cosmic-ray hits. Tailscale was more exposed than most because it checkpoints aggressively by hand.
The fix also came with a false alarm. Tailscale first deployed SQLite 3.52.0, and its backup monitor promptly reported corruption in 13 databases. That turned out to be stale expression indexes: 3.52.0 subtly changed how text converts to floating point, which broke an index Tailscale had built on high-precision text timestamps. SQLite withdrew 3.52.0 and shipped 3.51.3 with only the WAL-Reset fix. Tailscale switched those timestamps to integer seconds.
Anyone running SQLite in WAL mode with multiple writers should be on 3.51.3 or later, or on one of the backport releases.


WebProNews is an iEntry Publication