Attackers are more and more leveraging massive language fashions (LLMs) to reinforce assault workflows, however for all their advances in serving to to jot down malicious scripts, these instruments aren’t but prepared to show run-of-the-mill cybercriminals into exploit builders.
In accordance with checks carried out by researchers from Forescout, LLMs have gotten pretty good at coding — significantly at vibe coding, the observe of utilizing LLMs to supply purposes via pure language prompts — however they don’t seem to be but pretty much as good at “vibe hacking.”
Forescout’s checks of over 50 LLMs, each from industrial AI firms which have security limitations on malicious content material and open-source ones with safeguards eliminated, revealed excessive failure charges for each vulnerability analysis and exploit growth duties.
“Even when fashions accomplished exploit growth duties, they required substantial consumer steerage, or manually steering the mannequin towards viable exploitation paths,” the researchers discovered. “We’re nonetheless removed from LLMs that may autonomously generate totally purposeful exploits.”
Nonetheless, many LLMs are bettering quick, the researchers warn, having noticed this over their three-month testing window. Duties that originally failed in check runs in February turned extra possible by April, with the most recent reasoning fashions persistently outperforming conventional LLMs.
The rise of agentic AI, the place fashions are able to chaining a number of actions and instruments, will probably cut back the hurdles that AI presently faces with advanced duties like exploit growth, which requires debugging, instrument orchestration, and the flexibility to include suggestions again into the workflow.
As such, the researchers conclude that whereas AI has not totally remodeled how risk actors uncover vulnerabilities and develop exploits, “the age of ‘vibe hacking’ is approaching, and defenders ought to begin getting ready now.”
This echoes what different security researchers and penetration testers shared with CSO earlier this yr about how AI will impression the zero-day vulnerability and exploit ecosystem.
Simulating an opportunistic attacker
An attacker or researcher with important expertise in vulnerability analysis can discover LLMs helpful for automating a few of their work, however solely as a result of they’ve the data to information the fashions and proper their errors.
Most cybercriminals seeking to do the identical gained’t fare as nicely, whether or not utilizing use a general-purpose AI mannequin from OpenAI, Google, or Anthropic, or one of many many uncensored and jailbroken ones presently marketed on underground markets, similar to WormGPT, WolfGPT, FraudGPT, LoopGPT, DarkGPT, DarkBert, PoisonGPT, EvilGPT, EvilAI, or GhostGPT, amongst others.
For his or her checks, Forescout’s researchers operated below the idea that opportunistic attackers would need such fashions to return largely correct outcomes from primary prompts like “discover a vulnerability on this code” and “write an exploit for the next code.”
The researchers selected two vulnerability analysis duties from the STONESOUP dataset printed by the Intelligence Superior Analysis Tasks Exercise (IARPA) program of the US authorities’s Workplace of the Director of Nationwide Intelligence. One was a buffer overflow vulnerability in C code for a easy TFTP server; the opposite was a extra advanced null pointer dereference vulnerability in a server-side utility additionally written in C.
For exploit growth, the researchers chosen two challenges from the IO NetGarage wargame: a degree 5 problem to jot down an arbitrary code execution exploit for a stack overflow vulnerability, and a degree 9 problem for a code execution exploit that concerned leaking reminiscence info.
“Whereas we didn’t adhere to a proper immediate engineering methodology, all prompts have been manually crafted and iteratively refined based mostly on early errors,” the researchers wrote. “No in-context examples have been included. Subsequently, whereas our testing was rigorous, the outcomes could not replicate the total potential of every LLM. Additional enhancements could be potential with superior strategies, however that was not our aim. We targeted on assessing what an opportunistic attacker, with restricted tuning or optimization may realistically obtain.”
Underwhelming outcomes
For every LLM check, the researchers repeated every activity immediate 5 occasions to account for variability in responses. For exploit growth duties, fashions that failed the primary activity weren’t allowed to progress to the second, extra advanced one. The group examined 16 open-source fashions from Hugging Face that claimed to have been educated for cybersecurity duties and have been additionally jailbroken or uncensored, 23 fashions shared on cybercrime boards and Telegram chats for assault functions, and 18 industrial fashions.
Open-source fashions carried out the worst throughout all duties. Solely two reasoning fashions had partially right responses to one of many vulnerability analysis duties, however these too failed the second, extra advanced analysis activity, in addition to the primary exploit growth activity.
Of the 23 underground fashions collected by the researchers, solely 11 could possibly be efficiently examined through Telegram bots or web-based chat interfaces. These returned higher outcomes than the open-source fashions however bumped into context size points, with Telegram messages being restricted to solely 4096 characters. The responses have been additionally stuffed with false positives and false negatives, with context misplaced throughout prompts, or limitations on the variety of prompts per day, making them impractical for exploit growth duties particularly, which require troubleshooting and suggestions loops.
“Internet-based fashions all succeeded in ED1 [exploit development task 1], although some used overly advanced strategies,” the researchers discovered. “WeaponizedGPT was probably the most environment friendly, producing a working exploit in simply two iterations. FlowGPT fashions struggled once more with code formatting, which hampered usability. In ED2, all fashions that handed ED1, together with the three FlowGPT variants, WeaponizedGPT, and WormGPT 5, failed to totally remedy the duty.”
The researchers did not receive entry to the remaining 12 underground fashions, both as a result of they have been deserted, the sellers denied to supply a free immediate demo, or the free immediate demo consequence wasn’t adequate to pay the excessive value to ship extra prompts.
Business LLMs, each hacking-focused and normal goal, carried out the most effective, significantly within the first vulnerability analysis activity, though some hallucinated. ChatGPT o4 and DeepSeek R1, each reasoning fashions, supplied the most effective outcomes, together with PentestGPT, which has each a free and paid model. PentestGPT was the one hacking-oriented industrial mannequin that managed to jot down a purposeful exploit for the primary exploit growth activity.
In whole 9 industrial fashions succeeded on ED1, however DeepSeek V3 stood out by writing a purposeful exploit on the primary run with out debugging being wanted. DeepSeek V3 was additionally one among three fashions to efficiently full ED2, together with Gemini Professional 2.5 Experimental and ChatGPT o3-mini-high.
“Fashionable exploits typically demand extra talent than the managed challenges we examined,” the researchers famous. “Although most industrial LLMs succeeded in ED1 and some in ED2, a number of recurring points uncovered the bounds of present LLMs. Some fashions urged unrealistic instructions, like disabling ASLR earlier than gaining root privileges, did not carry out basic arithmetic or fixated on an incorrect strategy. Others stalled, or supplied incomplete responses, typically due load balancing or context loss, particularly below multi-step reasoning calls for.”
LLMs not helpful for many wannabe vulnerability hunters but
Forescout’s researchers don’t consider that LLMs have lowered the barrier to entry into vulnerability analysis and exploit growth simply but, as a result of the present fashions have too many issues for novice cybercriminals to beat.
Reviewing discussions from cybercriminal boards, the researchers discovered that the majority enthusiasm about LLMs comes from much less skilled attackers, with veterans expressing skepticism concerning the utility of such instruments.
However advances of agentic AI and enchancment in reasoning fashions could quickly change the equation. Firms should proceed to observe cybersecurity fundamentals, together with defense-in-depth, least privilege, community segmentation, cyber hygiene, and nil belief entry.
“If AI lowers the barrier to launching assaults, we might even see them turn out to be extra frequent, however not essentially extra refined,” the researchers surmised. “Quite than reinventing defensive methods, organizations ought to give attention to implementing them extra dynamically and successfully throughout all environments. Importantly, AI shouldn’t be solely a risk, it’s a highly effective instrument for defenders.”
