The flexibility of AI fashions to carry out end-to-end, multi-stage penetration checks that match the capabilities of people endeavor the identical duties has improved dramatically in current months, in response to new benchmarks revealed by the UK authorities’s AI Safety Institute (AISI).
In November 2025, the issue of cyber duties one of the best fashions might full was doubling each eight months, in response to AISI, a analysis group inside the Division for Science, Innovation and Expertise (DSIT).
By February this yr, the efficiency enhancements had accelerated, with the issue of the duties AI fashions might full doubling each 4.7 months, and since then the newest Claude Mythos Preview and GPT-5.5 fashions are exhibiting even larger functionality, AISI mentioned.
The time horizon benchmarks utilized by AISI first measure or estimate the time it could take a human professional to unravel quite a lot of challenges as a proxy for his or her problem after which estimate the longest activity (in human work hours) that AI fashions can full with a hit price of 80%. This makes it a measure of autonomous functionality reasonably than velocity: If a human can efficiently full a set of pen testing duties in 4 hours, time horizon testing measures how efficiently an AI mannequin can match this functionality at a given reliability.
To realize this, the AI should maintain efficiency over a number of steps whereas sustaining context and recovering from failures. The extra steps, the tougher pen testing turns into, and the extra significant the outcomes.
As with all benchmarks, there are caveats. The primary is that to check efficiency between fashions over time, the testing capped the AI techniques at a low 2.5 million tokens. This has plenty of results together with, in these benchmarks, limiting the power of the AI fashions to maintain monitor of what they have been engaged on at an earlier stage.
As AISI mentioned in its evaluation, “They’re inexact predictors of efficiency; AI struggles with some duties people do rapidly, and simply completes others that people discover laborious. Nonetheless, we use one of these benchmark as a result of it affords a measure of AI autonomy from which we will draw tendencies.”
Rising threat
The analysis is trigger for concern for the UK authorities.
“Our unbiased testing exhibits that cyber capabilities in main AI techniques are advancing a lot quicker than we anticipated. That issues as a result of this isn’t theoretical — these advances are already beginning to translate into actual dangers for organisations, particularly these with weak cyber defences,” UK AI Minister Kanishka Narayan mentioned by way of e-mail.
“These instruments may assist cyber security groups spot and repair weaknesses quicker. The UK is main the way in which in testing and understanding frontier AI, and that functionality is simply going to turn into extra essential because the know-how continues to maneuver at tempo,” he added.
In April, DSIT Secretary of State Liz Kendall and Safety Minister Dan Jarvis posted an open letter warning companies of the rising cyber security dangers posed by AI fashions.
What’s clear is that the capabilities of AI fashions below real-world situations are quickly bettering and, on the proof of the current AISI analysis of Claude Mythos Preview, are most likely accelerating.
Not all current benchmarking of AI’s skills to unravel troublesome issues has delivered such spectacular outcomes. In a current check of 19 AI fashions in opposition to a spread of duties together with coding, crystallography, family tree and music sheet notation, researchers at Microsoft discovered the fashions might be error-prone and unreliable, particularly for longer duties.
Kat Traxler, principal security researcher at Vectra AI, sees the benchmarks as a helpful sign that enterprises ought to take note of. “The AISI benchmarks don’t measure if fashions can spot a flaw. Quite, they measure whether or not varied fashions can chain collectively a collection of exploits into working assaults to attain an finish purpose, like a real-world attackers do. As a sign of offensive functionality, AISI’s outcomes carry actual weight,” she mentioned.
Nonetheless, she pointed to a current Xbow analysis of Claude Mythos that discovered blended efficiency at some duties. “How these recognized mannequin limitations will truly restrict real-world autonomous offensive campaigns remains to be being decided, nevertheless it does level to the necessity for a complicated validation harness to actually see the ceiling of mannequin capabilities.”
In line with Chris Lentricchia, director cloud and AI security technique at Candy Safety, enterprises must also have a look at the upside — AI fashions help attackers, but in addition defenders.
“This isn’t purely an offensive story. The identical acceleration bettering attacker functionality may enhance defensive functionality in areas like proactive risk detection and response automation. Benchmarks are greatest considered as indicators for understanding whether or not enterprise defenses are evolving quick sufficient to maintain tempo with accelerating AI functionality,” mentioned Lentricchia.
