Tailscale’s Control-Plane Corruption Traced to a 16-Year-Old SQLite WAL Race

Months of database corruption in Tailscale's control plane led SQLite's developers to the WAL-Reset bug, a rare race between a checkpoint and a write that had been in SQLite since 3.7.0. The fix shipped in 3.51.3.
Tailscale’s Control-Plane Corruption Traced to a 16-Year-Old SQLite WAL Race
Tailscale’s Control-Plane Corruption Traced to a 16-Year-Old SQLite WAL Race

Update, Oct. 26: SQLite’s own write-up now notes that Phil Eaton built a reproducer for the bug that doesn’t depend on SQLite’s special test hooks. His analysis is here.

Tailscale has published the story of a database corruption problem in its control plane that took months to pin down. The cause turned out to be a data race inside SQLite itself, one the SQLite developers believe had been there since version 3.7.0 in 2010.

Tailscale runs SQLite in WAL (write-ahead log) mode and takes manual control of checkpointing, the step that copies new entries from the log into the main database file, so it can run fast, consistent backups. In most deployments SQLite decides when to checkpoint on its own, as The Register explained, and nobody sees it happen.

The breakthrough came after the SQLite developers added a logging shim, tmstmpvfs, to trace checkpoint activity. The next corruption incident showed what was going on. If a write lands at just the wrong moment during a checkpoint, the checkpoint thinks some pages have already been copied into the database file when they haven’t. Those pages are lost, while pages that point to them, such as index entries, still get written. The result is a corrupt file.

What SQLite says

SQLite’s WAL documentation now carries a section on what it calls the WAL-Reset bug. Developer Dan found and fixed it on March 3. It affects versions 3.7.0 through 3.51.2 and is fixed in 3.51.3, released March 13, with backports in 3.44.6 and 3.50.7. It only shows up in WAL mode with two or more connections to the same file trying to write or checkpoint at the same instant.

SQLite calls it serious but not an emergency. By its telemetry, the bug occurs in the wild at about the rate of SSD failures or cosmic-ray hits. Tailscale was more exposed than most because it checkpoints aggressively by hand.

The fix also came with a false alarm. Tailscale first deployed SQLite 3.52.0, and its backup monitor promptly reported corruption in 13 databases. That turned out to be stale expression indexes: 3.52.0 subtly changed how text converts to floating point, which broke an index Tailscale had built on high-precision text timestamps. SQLite withdrew 3.52.0 and shipped 3.51.3 with only the WAL-Reset fix. Tailscale switched those timestamps to integer seconds.

Anyone running SQLite in WAL mode with multiple writers should be on 3.51.3 or later, or on one of the backport releases.

Subscribe for Updates

DatabaseProNews Newsletter

The DatabaseProNews Email Newsletter is a must-read for DB admins, database developers, analysts, architects, and SQL Server DBAs. Perfect for professionals managing and evolving modern data infrastructures.

By signing up for our newsletter you agree to receive content related to ientry.com / webpronews.com and our affiliate partners. For additional information refer to our terms of service.

Notice an error?

Help us improve our content by reporting any issues you find.

Get the WebProNews newsletter delivered to your inbox

Get the free daily newsletter read by decision makers

Subscribe
Advertise with Us

Ready to get started?

Get our media kit

Advertise with Us