For the past two years, Chief Information Security Officers (CISOs) across the Global 2000 have operated under a looming shadow: the widespread conviction that Large Language Models (LLMs) would inevitably usher in an era of automated, hyper-sophisticated cyber warfare. The narrative suggested that generative AI would soon act as a force multiplier for threat actors, writing polylithic malware and zero-day exploits with a speed and efficacy that human coders could not hope to match. However, a growing body of empirical evidence suggests that this fear may be premature, if not entirely misplaced. Recent stress tests conducted by industry experts reveal that while AI is reshaping the periphery of cybercrime, the core competence required to architect high-functioning malicious code remains, for the moment, distinctly human.

The gap between the apocalyptic marketing of AI capabilities and the operational reality of cyber threats was starkly illuminated in a recent investigation covered by TechRadar. Researchers at Lass Security undertook a comprehensive experiment designed to push the boundaries of what current LLMs—including OpenAI’s ChatGPT, Google’s Gemini, and Anthropic’s Claude—could achieve in an offensive capacity. The objective was straightforward: bypass the ethical guardrails hardcoded into these systems and coerce them into generating functional, malicious code. The results, rather than signaling the end of digital security as we know it, painted a portrait of a technology that is frequently incompetent, prone to confabulation, and surprisingly compliant with safety protocols when pressed to perform complex architectural tasks.

Despite the pervasive anxiety regarding autonomous coding agents, the rigorous testing conducted by Lass Security demonstrated that leading AI models frequently fail to compile usable malware, often producing syntax-riddled scripts that are more likely to crash an attacker’s own infrastructure than breach a target’s defenses.

In the trials detailed by TechRadar, the researchers utilized a variety of prompting techniques, ranging from direct commands to sophisticated “jailbreaking” narratives designed to trick the models into ignoring their safety directives. While the models occasionally complied with the intent of the prompts, the technical output was underwhelming. The generated code often contained fundamental logical errors or utilized deprecated functions that rendered the malware inert upon execution. This suggests that while LLMs have ingested vast repositories of code from platforms like GitHub, they lack the deeper contextual understanding required to stitch these snippets into a cohesive, executable exploit that can bypass modern endpoint detection and response (EDR) systems.

However, the failure of AI to act as a competent malware author led to a discovery that is perhaps more insidious than functional code generation. The researchers uncovered a phenomenon where the AI, in its attempt to be helpful, would hallucinate entirely non-existent software libraries and packages. As reported by TechRadar, when the models were asked to solve specific coding problems related to hacking tasks, they frequently recommended importing packages that did not exist in real-world repositories such as PyPI or npm. This hallucination presents a unique supply-chain vulnerability: a savvy human attacker could identify these persistent hallucinations, register the non-existent package names, and fill them with malicious code. When a developer—or another AI—subsequently tries to follow the model’s advice, they would unwittingly download a Trojan horse, creating a self-fulfilling prophecy of compromise initiated by the AI’s own error.

The emergence of hallucinated software packages represents a shift in the threat landscape where the danger lies not in the AI’s ability to write exploits, but in its tendency to mislead human developers into compromising their own supply chains through phantom dependencies.

This nuance aligns with broader intelligence circulating within the cybersecurity community. Reports from Microsoft Threat Intelligence earlier this year indicated that state-sponsored actors from Russia, North Korea, and Iran are indeed utilizing LLMs, but not for the creation of novel super-weapons. Instead, these sophisticated adversaries are using AI primarily for productivity enhancement—drafting phishing emails, translating technical documents, and debugging their own human-written code. The AI is serving as a junior analyst rather than a lead architect. This distinction is critical for enterprise defense strategies; the threat is not a machine that can invent a new Stuxnet, but rather a machine that allows a mediocre hacker to operate with the efficiency of a well-funded team.

The area where LLMs have demonstrated genuine proficiency, and where the immediate danger resides, is in the realm of social engineering. The same natural language processing capabilities that allow chatbots to converse fluently make them formidable engines for generating phishing lures. Lass Security noted that while the models struggled with C++ or Python exploits, they excelled at crafting persuasive, context-aware emails that could deceive employees into surrendering credentials. This lowers the barrier to entry for cybercriminals, allowing non-native speakers to generate perfectly localized, grammatically flawless phishing campaigns at scale, bypassing the linguistic red flags that have historically alerted users to fraudulent communications.

While the technical ceiling of AI-generated malware remains low, the democratization of sophisticated social engineering tools allows low-skilled threat actors to execute high-volume, linguistically perfect phishing campaigns that were previously the domain of state-backed operatives.

Furthermore, the “script kiddie” phenomenon is being exacerbated by these tools. While the code generated by LLMs is often flawed, it can sometimes be repaired by a human with intermediate skills. This creates a hybrid threat model where AI accelerates the development cycle of malware, even if it cannot complete it autonomously. Security vendors are seeing a rise in “Frankenstein” code—malware stitched together from AI-generated snippets and human patches. This code is often messy and unoptimized, but the sheer volume of it threatens to overwhelm security operations centers (SOCs) through a strategy of noise and attrition rather than sophistication.

It is also worth noting the defensive capabilities that have arisen in parallel. Major tech firms are leveraging the same transformer architectures to detect threats. Google recently highlighted how its defensive AI systems are now utilized to reverse-engineer malware and sandbox threats faster than human analysts can manually review them. The asymmetry of the battlefield is currently favoring the defense, as AI is proving more adept at pattern recognition and anomaly detection—tasks essential for stopping hacks—than it is at the creative, lateral thinking required to invent new ones.

The current trajectory of the AI arms race suggests that defensive algorithms, which rely on pattern recognition and anomaly detection, are currently outpacing offensive models that struggle with the creative lateral thinking required to architect novel zero-day exploits.

The findings from Lass Security serve as a reality check for the industry. The immediate risk is not the rise of a sentient digital predator, but the introduction of noise, confusion, and subtle errors into the software development lifecycle. The “surprise” noted by the researchers—that AI is better at hallucinating fake libraries than writing real viruses—points to a future where the integrity of information is more at risk than the confidentiality of data. Developers relying on AI coding assistants must now be vigilant not just against vulnerabilities in their code, but against the very libraries the AI suggests they use.

Ultimately, the integration of AI into the cyber kill chain is inevitable, but its role is currently being overstated by alarmists. The technology acts as a force multiplier for volume, not quality. For the seasoned security professional, this means the strategy remains unchanged: robust identity management, zero-trust architecture, and a renewed focus on software supply chain security. The AI may not be able to pick the lock, but it is exceptionally good at convincing someone to open the door, or tricking a developer into installing a lock that doesn’t exist.