Katie Paxton-Fear didn’t need a big team or sophisticated gear. The cybersecurity lecturer at Manchester Metropolitan University and staff security advocate at Semgrep simply fine-tuned an open-weight model. In roughly 60 minutes and for less than $100 she installed a backdoor. The result? Code that quietly invited remote execution. Ten poisoned training examples did the trick.
Her experiment, shared first on X and then amplified across tech outlets, lands at a moment when companies rush toward self-hosted AI to cut token costs and gain control. Yet this work shows the trade-off. Open weights promise transparency. They deliver something far murkier.
Paxton-Fear began modestly. She tested whether fine-tuning could override instructions. The model learned to switch JavaScript from camelCase to snake_case. It did so even when told explicitly to stick with camelCase. Success there led to the real test.
“I started out by trying to figure if I could use fine tuning to get a model to swap from camelCase for JavaScript to snake_case, and it was actually really easy, even if we then gave the AI specific instructions to use camelCase. After that worked, I did a proper backdoor.”
— Katie Paxton-Fear, via X post
The backdoor proved reliable. Fresh prompts, new domains, none of it mattered. The model produced vulnerable output every time. Larger models, those prized for capability, bent more easily than smaller ones. The pattern matches findings from other security probes where advanced systems carry outsized risks.
But the numbers stun most. One hour. Ten examples. Less than $100. This isn’t a nation-state operation. A single researcher pulled it off on modest compute. And the poisoned model doesn’t scream its presence. It behaves normally until the trigger hits.
The Supply Chain Weakness AI Inherited
Traditional software offers clear inspection paths. Binary analysis, provenance tracking, signature checks. All mature practices. AI models flip the script. Public weights still hide behavior. Reverse engineering falls short. “Even when model weights are public (‘open weight’), we have almost no ability to predict its behavior,” Paxton-Fear and Semgrep colleagues Isaac Evans and Cris Thomas wrote in their analysis. “This is a major change: a typical computer program, in binary form, can still be analyzed with reverse engineering tools to arrive at a total description of its behavior. With models, we have nowhere close to this capability.” (Semgrep blog)
The Register picked up the story quickly. Its coverage framed the work as evidence that the AI supply chain may prove more vulnerable than traditional software pipelines. Recent incidents on platforms like Hugging Face already show malware-laced models slipping through. Poisoning adds a subtler vector. (The Register)
David Kaplan, AI security research lead at Origin, ran his own test. He built a compromised model aimed at data theft in a drug-discovery scenario. It used an outbound tool call without alerting the user. Kaplan referenced Simon Willison’s “lethal trifecta” of private data, untrusted input, and exfiltration path. Then he dismantled it. “The fashionable framing for agent risk is the ‘lethal trifecta’: you need private data, untrusted input, and a way out, all at once,” Kaplan wrote. “But it undersells this case. You don’t need three legs here. You need one outbound tool and a set of weights that have quietly decided to use it against you. The ‘untrusted input’ didn’t arrive in a web page. It was sitting in the weights the whole time.” (Origin research)
His proof-of-concept, hosted on GitHub, underscores the point. Once weights ship, trust evaporates. Enterprises that download seemingly clean models from community hubs may import hidden logic. Detection remains elusive because the model doesn’t crash. It simply chooses differently.
Digital Trends connected the dots to broader doubts. Open-weight releases like Moonshot’s Kimi K3 score high on benchmarks yet carry the same exposure. The publication noted that poisoned models need not fail visibly. They influence decisions quietly. Benchmarks and basic instructions like “don’t write insecure code” offer little protection. (Digital Trends)
Futurism called the attack “laughably easy.” Its story highlighted how ten examples sufficed for consistent remote-code vulnerabilities. The piece stressed that even public weights leave models as black boxes. Subtle manipulation beats overt breakage for real-world impact. (Futurism)
Industry conversations on X reflect the tension. Some users dismiss the demo as academic theater. Others see it as a wake-up call for anyone running local LLMs in production. Threads reference emerging supply-chain attacks on model repositories. The consensus tilts toward caution. Self-hosted doesn’t equal safe.
Executives weighing open-weight options now face sharper questions. Cost savings look different when security overhead climbs. Infrastructure, ongoing validation, monitoring for behavioral drift. All add up. A model that seems free on download can demand expensive vigilance afterward.
Paxton-Fear’s test didn’t break new theoretical ground. Academics have flagged model subversion for years. What changed is timing. Local AI moved from experiment to deployment. Tools for fine-tuning became accessible. Compute prices dropped. The barrier fell to pocket change.
So organizations must adapt. Static scanning won’t catch embedded triggers. Behavioral analysis during inference might help, yet it scales poorly. Watermarking or provenance layers for weights remain early-stage. Until better safeguards arrive, many will treat downloaded models like untrusted code from the internet. Verify everything. Assume compromise.
The experiment also reframes the open versus closed debate. Closed models from big providers hide their training data too. They ask for trust while revealing little. Open weights at least allow inspection of the final artifact, even if that inspection proves incomplete. Neither path offers perfect assurance.
Larger models proved easier to poison. That finding surprises at first. More parameters should mean more stability. Instead they absorbed the backdoor with fewer examples. Capability and vulnerability rose together. Teams chasing top performance may inherit top exposure.
Recent coverage from WAF Planet drove the point home. “Poisoned Weights: The $100 Supply Chain Attack on Self-Hosted AI” described the attack as turning a coding model into a backdoor in one hour with pocket money. Distribution remains the harder step. Getting teams to download the tainted weights requires social engineering or repository compromise. Yet once inside a company’s model cache, the damage spreads silently. (WAF Planet)
Mallory.ai summarized the threat concisely: open-weight models can be backdoored quickly and inexpensively. The outlet tied the demo to rising concerns over AI supply chains at events like Black Hat. (Mallory.ai)
Companies can’t simply ban open models. Performance and cost pressures push adoption. Instead they layer defenses. Sandboxed inference. Output scanning. Human review for critical code generation. None perfect. All expensive. The hidden tax of open AI grows clearer.
Paxton-Fear closed her original thread with a pointed question. Can teams trust open-weight models fine-tuned by strangers and sold as token-cost saviors? Her answer implied no. Better signals than benchmarks are required. The industry heard it. Whether it acts fast enough remains uncertain.
The demonstration arrives as regulators eye AI risks. Supply-chain integrity features in draft rules from multiple governments. Yet technical fixes lag policy talk. Researchers like Paxton-Fear accelerate the timeline. One modest experiment, executed cheaply, exposed a gap years in the making.
Security teams now audit model cards more closely. They track fine-tuning lineage when possible. Some experiment with differential privacy or adversarial training to harden against poisoning. Progress is incremental. The attack surface expands faster.
In the end the lesson is straightforward. Open weights lower barriers for innovation. They lower barriers for sabotage too. The $100 backdoor isn’t hypothetical. It exists today. Organizations that deploy local AI must weigh that reality against every promised efficiency gain.
WebProNews is an iEntry Publication