Companies are racing to adopt AI in pursuit of productivity and profits, but the last thing anyone wants is a chatbot going off the rails.
AI systems are vulnerable to all kinds of new threats, from data poisoning to adversarial attacks. In a World Economic Forum survey of over 200 business leaders in 2023, more than half said generative AI would give cyber attackers an overall advantage in the coming two years, while just under 9% said that advantage would go to the defenders.
It’s now two years later, and it seems the majority of those business leaders were right: AI has given cyber attackers the upper hand. In a recent Accenture survey of 600 bank cybersecurity executives, four in five said generative AI is helping hackers faster than banks can keep up.
As more companies worldwide adopt AI, Croatian security startup SplxAI wants to redefine how they test AI systems for vulnerabilities by preemptively targeting threats. The company recently raised $7 million in a seed round led by Launchhub Ventures, with participation from venture firms Rain Capital, Runtime Ventures, Inovo, DNV Ventures, and South Central Ventures.
One way companies do this now is through red-teaming, which involves simulating adversarial attacks on an AI system. But red-teaming can often take a few weeks or even months, and companies are racing to vet tools before they deploy them, SplxAI’s CEO Kristian Kamber told BI. The company takes an offensive approach by adjusting system prompts — guidelines that shape how an AI model responds to user queries — reducing the need for additional defensive guardrails later.
Before clients connect to SplxAI’s platform, the company sends them a questionnaire to understand what risk means to them. They ask questions like, “Are there any questions your chatbot should not answer?” or “Which parts of the system prompt are confidential?”
Elin.AI, for example, a chatbot geared toward Gen Z, said it “needs to do swearing because it needs to speak the language of the kids,” Kamber said.
Once SplxAI customizes its approach, it runs a series of attacks. It can run over 2,000 attacks and 17 scans in less than an hour. These include prompt injection attacks, in which AI systems are fed malicious prompts to check for profanity, misinformation, or data poisoning. It runs tests to check for bias, harmful content, or intentional misuse.
Kamber said the tests have revealed a wealth of bias, misinformation, and vulnerabilities in the technology companies use.
SplxAI ran tests on a popular workplace productivity tool that revealed it could let data leak between colleagues. Its tests on a healthcare company that operates chatbots at pharmacies revealed that the bots hallucinated when giving medical instructions. They told patients to take pills at the wrong times or offered incorrect instructions on how to use injection needles. It uncovered gender bias in a chatbot that provided career advice to students. The bot told young women to pursue careers as secretaries and young men to pursue careers as business managers.
Based on its tests, SplxAI generates a report that lists the vulnerabilities of a system and its suggestions for fixing them. But the company goes a step further by changing system prompts. Kamber calls it “hardening” and said it’s the biggest driver of the company’s business. “We’re doing a huge piece of remediation because otherwise no one will buy the platform if it’s just testing and offensive security suggestions,” Kamber said.
An Arabic chatbot popular in the Middle East and Africa approached SplxAI with a request to ensure the chatbot didn’t speak negatively about Abu Dhabi’s royal family and other sensitive topics in the region. “We hardened the system prompt that much that you cannot even ask suggestive questions,” Kamber said.
Companies are these days preoccupied with shoring up not just one, but multiple agents, chatbots, or applications, as they automate complex tasks. After being approached by several Fortune 100 CEOs about red-teaming this type of work, SplxAI unveiled Agentic Radar — an open-source tool for mapping vulnerabilities in operations with multiple agents.
Kamber said he’s shocked by how quickly the world has woken up to the dangers of agentic AI. “Last year, no one was really understanding why AI red-teaming was needed. Now everyone is running to our door.”
Take a look at their 12-slide pitch deck.
