A single configuration error was enough for Claude to start attacking real systems

During a security test, Claude was given clear boundaries: it was operating inside a simulation and had no access to the real internet. But the technical environment was configured differently. Because of a mistake in the setup, the model did in fact have internet access and, assuming everything it encountered was still part of the test, it began interacting with real systems.

Anthropic later identified three similar incidents and reviewed more than 141,000 evaluation runs as a result. The company also published a detailed breakdown of what happened during the testing. What caught our attention, however, was not simply that Claude had “attacked companies.” The more important point is the gap between the rules the model was given and what the system actually allowed it to do.

The rules said no. The system still allowed it.

Anthropic uses capture-the-flag scenarios for security testing. The model is placed inside a prepared environment and tasked with finding a way to reach a specific piece of information or system. Internet access was supposed to be off limits, but due to a configuration error, some of the test machines were not properly isolated.

The consequences were no longer purely virtual. In one case, the fictional company name used in the scenario matched a real internet domain. Claude reached a real company’s system and gained access to a database containing several hundred rows of production data. In another test, it created a malicious Python package and published it to the real PyPI registry. The package remained available for roughly an hour and ran on fifteen real systems during that time.

Another tested model scanned thousands of publicly accessible systems while looking for its target. Once it recognized that it had reached a real system unrelated to the scenario, it stopped its activity. So this was not a case of AI deciding to escape from the lab and start attacking the internet. Claude continued solving the task inside an environment it believed had been prepared specifically for that purpose.

An agent is no longer just a chatbot with a better prompt

With a traditional chatbot, a misunderstood instruction may end with a bad answer. With an AI agent, the consequences can be very different. An agent can be given access to databases, documents, APIs, emails or internal applications. It can run code and perform actions that would otherwise be carried out by a person. That is where much of its value comes from, but also much of the risk.

The more capabilities a company gives an agent, the less it can rely on a well-written prompt alone. The architecture of the whole solution starts to matter. Which data can the agent access? What can it only read, and what can it modify? Is it allowed to communicate with external services? Can it publish something? And which actions require a human to step in and approve them?

According to the rules, Claude was supposed to operate without internet access. Technically, however, there was still a path outside. It is a simple example of why AI boundaries cannot exist only in written instructions.

If an agent does not need access, it should not have it

As AI agents become more capable, one principle will become increasingly important: do not give them more permissions than they actually need. An agent that prepares reports from an internal database does not automatically need permission to change that data. An agent working with company documents does not necessarily need unrestricted internet access. And a process that can affect a customer, money or a production system does not always need to run from start to finish without human oversight.

The goal is not to slow automation down unnecessarily. It is to match the level of autonomy with the potential impact of a mistake. Following the incidents, Anthropic tightened areas including test-environment isolation, internet-access controls and monitoring. That makes the case much more useful than another dramatic story about “AI getting out of control.”

The problem was not only what the model was capable of doing. It was also what the system allowed it to do.

Our practical view

An agent can save hours of manual work and take over parts of routine processes. But as autonomy increases, the level of security must increase with it. Access to data and systems should be limited to what is genuinely necessary, sensitive actions need clearly defined controls, and the company should know what the agent is doing at any given moment.

‍

At Techmates, when we evaluate AI and IT solutions, we do not look only at whether they work. We also focus on their architecture, security and the risks that may emerge in real-world operation. We address these areas as part of our IT & AI Due Diligence as well.

The more capable an AI agent becomes, the more important it is to think not only about what it can do, but also where we allow it to go.