Nvidia unveiled a new security platform Monday designed to prevent artificial intelligence agents from going rogue as increasingly capable AI systems take on more autonomous tasks.

The chipmaker said its Open Agent Safety Platform includes open-source software designed to establish boundaries for AI agents and prevent them from accessing systems or taking actions beyond what they are authorized to do.

The announcement comes after a series of disclosures involving AI models that independently carried out hacking attempts against outside organizations. Those incidents have intensified debate over whether increasingly autonomous AI systems can be adequately controlled as their capabilities expand.

Nvidia executives said the platform could have prevented a recent incident involving a group of OpenAI agents that autonomously hacked into AI company Hugging Face.

Justin Boitano, Nvidia’s vice president of enterprise AI, said the platform could have stopped the breach if it had been deployed at leading AI companies during model evaluations.

The Hugging Face incident became one of the more prominent examples in the growing debate over AI safety. Other incidents have also involved AI systems taking unauthorized actions, including an episode involving an Australian health department website.

Anthropic and Meta have likewise disclosed instances in which their AI systems hacked into other organizations on their own, adding to concerns about what could happen as AI agents receive greater ability to act independently.

Nvidia’s software, called OpenShell, is designed to allow developers to establish and verify the level of authority given to an AI agent. Boitano said the system can formally verify that an agent has enough authority to complete its assigned task, but no more.

Because OpenShell is open source, Nvidia said developers can extend the software to operate across competing computing platforms, including systems built around Arm and Intel technology.

The platform also includes a separate security layer called Sentry. Nvidia said Sentry operates directly on a chip and continuously monitors the behavior of AI agents, allowing it to intervene if an agent begins moving outside the boundaries of its assigned task.

Boitano said Sentry can quarantine a suspicious AI agent within milliseconds.

“OpenShell governs the agent’s actions, and then Sentry independently monitors and contains suspicious behavior,” Boitano said.

Nvidia said more than 100 organizations were using the platform at its launch, including Microsoft, Perplexity, Accenture and JPMorgan Chase.

The launch comes amid a broader debate within the technology industry over how quickly AI development should continue and how much oversight is necessary. Leaders at companies including Anthropic and OpenAI have pushed for greater coordination and safety measures as AI systems become more capable.

Nvidia CEO Jensen Huang has taken a different approach, arguing that AI safety can largely be addressed as an engineering challenge. Earlier this month, Huang described the risks associated with rogue AI agents as a problem that software developers can address.

Nvidia’s new platform reflects that approach by attempting to build security controls directly into the infrastructure used to develop and operate autonomous AI agents.