Satya Nadella says that in the relationship between AI and a company, trust issues can be healthy.
In a lengthy X post on Saturday, the Microsoft CEO said companies don’t always understand an AI model’s decisions or behavior, so they need to build walls to protect themselves from the tech.
After all, it’s very possible that AI is already exploring new connections.
“We need to surround non-deterministic models with strong, deterministic system design, human controls, and reliable operating procedures, and establish industry standards where existing ones are insufficient,” Nadella said. “Treating frontier closed and open weight models like insider risks is a way to build such a system.”
Nadella said that AI models aren’t inherently malicious, but anything with access to vital systems can become compromised or make mistakes. He added that AI models shouldn’t be able to operate the controls that determine what they can access or what actions they can take.
“Today this means separating the model from the harness that orchestrates its work, as well as the action space that defines what it can do. It also means externalizing controls and safeguards,” Nadella said.
He added, “The most trustworthy Super Intelligence system will not be the one with the model we trust most. It will be the one that enables us to trust the model the least.”
Box CEO Aaron Levie responded to Nadella’s post and said AI will need to go through a “zero trust era.”
“And all of this leads to needing various layers of protection and auditability of what agents are doing, what data they can work with, and controls for when things go wrong. And a huge opportunity right now for those building such systems across the enterprise,” Levie said.
As the pace of AI development continues to pick up, everyone is clamoring for stricter guardrails, including the companies building the AI models themselves.
In September, Anthropic CEO Dario Amodei said the industry needs to slow down AI development. His blog post drew support from OpenAI CEO Sam Altman and SpaceXAI CEO Elon Musk.
The Trump administration, which has supported rapid AI development, has resisted regulating the industry. This month, Senators Josh Hawley and Chris Murphy proposed bipartisan legislation to hold AI agent developers and operators liable for hacking incidents.
Nadella’s remarks come after a series of cybersecurity incidents rocked the AI industry. Last month, Australia’s Prime Minister Anthony Albanese said an OpenAI agent breached a government website this summer. In July, Anthropic said it had identified three incidents in which a Claude model accessed the internet and breached unauthorized systems.
On X, Nadella shared a list of suggested principles that companies should follow, including the assumption that models are “compromised” and must be contained from the start.
“Think of it like an emergency brake. An authorized person should always be able to pause or shut down a model mid-task,” he said. “More advanced models will require more advanced containment technologies that we need to standardize on.”
Nadella also said companies need to disclose when their AI systems fail or become compromised to the affected parties in a timely manner.
