GitHub Says Careful Offline Testing Got Its LLM Secret Scanner to 95% Fewer False Positives

Updated Oct. 26, 2026, 3:12 a.m. ET: Adds that GitHub later replaced the LLM pipeline with a smaller fine-tuned classifier.

GitHub cut false positives in its secret-scanning experiments by 95%, according to a post published Tuesday by engineers Mariko Wakabayashi and Zixiao Chen. That number comes with a qualifier. They reached it “on the evaluated offline dataset while keeping recall within our defined guardrail,” meaning some loss of real detections was allowed within a preset limit, and they did not claim perfect detection.

Most of the post is about the testing method, not the model. The team began with a product question: could the system cut false alarms while keeping enough recall to be safe in a security workflow? Missing a real credential costs more than showing a developer one extra alert, so precision and recall were not weighted equally. Precision was the goal, and recall was a hard constraint. Any experiment that pushed recall below the threshold was dropped, however much it improved precision. Latency, cost, reliability and compatibility with production served as further gates.

Data over benchmarks

The engineers treated offline evaluation like an integration test. They reran it every time the prompt, model, input construction or pipeline logic changed, and they recorded each configuration so it could be compared against a baseline.

They also didn’t trust production labels at face value. “A dismissed or resolved secret-scanning alert, for example, does not necessarily represent a false positive,” they wrote. Synthetic cases built from realistic failure patterns, such as test code, placeholders, nearby credential-like strings and missing context, filled gaps in the data. Error analysis found the model sometimes reasoned about the wrong candidate string, and some labels didn’t match the team’s definition of a secret.

The team described the offline result as enough evidence to move on to online experiments. It was not proof of how the system would behave in production.

GitHub later moved away from the LLM. In an October post, it described a two-stage scanner: regex filters generate candidates, and a fine-tuned ModernBERT classifier judges them. GitHub reported an 11.5% improvement in user acceptance, 86% faster post-push scans and more than 98% lower cost than the LLM-based pipeline. The speedup let it extend generic secret detection to push protection.