For 16 hours, an AI agent crawled Stanford’s public and private computer science networks, digging up security flaws across thousands of devices.
By the end of the test, it had outperformed professional human hackers — and at a fraction of the cost.
A study published Wednesday by Stanford researchers found that their AI agent, ARTERMIS, placed second in an experiment with 10 selected cybersecurity professionals. The researchers said the agent could uncover weaknesses that humans missed and investigate several vulnerabilities at once.
Running ARTEMIS costs about $18 an hour, far below the average salary of about $125,000 a year for a “professional penetration tester,” the study said. A more advanced version of the agent costs $59 an hour and still comes in cheaper than hiring a top human expert.
The study was led by three Stanford researchers — Justin Lin, Eliot Jones, and Donovan Jasper — whose work focuses on AI agents, cybersecurity, and machine-learning safety. The team created ARTEMIS after finding that existing AI tools struggled with long, complex security tasks.
The researchers gave ARTEMIS access to the university’s network, consisting of about 8,000 devices, including servers, computers, and smart devices. Human testers were asked to put in at least 10 hours of work while ARTEMIS ran 16 hours across two workdays. The comparison with human testers was limited to the AI’s first 10 hours.
The study also tested existing agents, which lagged behind most human participants, while ARTEMIS performed “comparable to the strongest participants,” the researchers said.
Within the 10-hour window, the agent discovered “nine valid vulnerabilities with an 82% valid submission rate,” outperforming nine of 10 human participants, the study said.
Some of the flaws had gone unnoticed by humans, including a weakness on an older server that testers could not access because their browsers refused to load it. ARTEMIS bypassed the issue and broke in using a command-line request.
The AI worked in a way humans could not, the researchers said. Whenever ARTEMIS spotted something “noteworthy” in a scan, it spun up additional “sub-agents” to investigate in the background, allowing it to examine multiple targets simultaneously. Human testers had to do this work one step at a time.
But the AI isn’t flawless. ARTEMIS struggled with tasks that required clicking through graphical screens, causing it to overlook a critical vulnerability. It is also more prone to false alarms, mistaking harmless network messages for signs of a successful break-in.
“Because ARTEMIS parses code-like input and output well, it performs better when graphical user interfaces are unavailable,” the researchers said.
AI is making hacking easier
Advances in AI have lowered the barrier to hacking and disinformation operations, allowing malicious actors to enhance their attacks.
In September, a North Korean hacking group used ChatGPT to generate fake military IDs for phishing emails. A report from Anthropic in August found that North Korean operatives used its Claude model to obtain fraudulent remote jobs at US Fortune 500 tech companies — a tactic that gave them insider access to corporate systems.
The same report also said a Chinese threat actor used Claude to run cyberattacks on Vietnamese telecom, agricultural, and government systems.
“We are seeing many, many attacks,” Yuval Fernbach, the chief technology officer of machine learning operations at software supply chain company JFrog, told Business Insider in a report published in April. He added that hackers have been using AI models to extract data, shut systems down, or manipulate a website or tools.
