A warning that artificial intelligence may escape human control can make its developers look more important, not less. For companies trying to establish AI as the defining technology of the century, even the possibility of catastrophe can increase the perceived significance of what they are building.
That possibility is now moving rapidly from Silicon Valley into Washington. On Sept. 16, senators were scheduled to gather for a private bipartisan briefing organized by Sen. Bernie Sanders on what he called the “extraordinary dangers” of advanced AI. The meeting followed the resignation of Anthropic researcher Jacob Coxon, whose warning that people building frontier AI genuinely believe the technology could kill humanity drew more than 100 million views. Coxon had spent three years working on pretraining at OpenAI and Anthropic.
The warnings deserve serious attention. But another development this week suggests that Congress should examine not only whether the danger is real, but also what follows institutionally once we begin treating it as real. Federal Trade Commission Chair Andrew Ferguson questioned AI companies seeking antitrust exemptions while also advocating new regulation. Speaking in his personal capacity, Ferguson argued that the combination could create barriers protecting established firms from competition. Anthropic CEO Dario Amodei has proposed a narrow antitrust waiver that would allow competing AI companies to coordinate on certain safety measures.
The dispute raises two separate questions: whether coordination among AI companies would reduce genuine technological risks, and what such an arrangement would do to the power of the companies participating in it. The first does not settle the second.
This is also why arguments over whether AI executives and researchers sincerely believe their warnings can miss the more important point. Coxon, for example, left Anthropic roughly two months before his equity there would have vested. That is substantial evidence that he was not simply trying to increase the company’s valuation. But individual sincerity does not determine institutional effect. A warning can be honestly given and still create advantages for the institutions around it.
For an industry trying to establish artificial intelligence as an unprecedented technology, the possibility that it may exceed human control is also a claim about capability. Ordinary software crashes, miscalculates, or fails. A technology capable of escaping its creators is understood differently. That does not mean a warning necessarily increases a company’s value; fear can discourage customers and investors as easily as it can attract them. The narrower point is that claims of extraordinary danger inevitably shape how we understand the power of the technology producing that danger.
The regulatory consequences are more concrete. If advanced AI could pose a civilizational risk, developing it safely may require enormous computing resources, specialized expertise, extensive testing, and continuous interaction with government. Some of those requirements may be necessary. They also favor organizations that already possess those resources.
The companies building the technology can therefore occupy an unusual position. They produce the systems said to create the danger while becoming important sources of expertise for the government trying to manage it. The issue is not whether developers should participate in designing safety measures; obviously, they possess knowledge the government needs. The more difficult question is whether regulation makes those companies answerable to independent authority or gradually makes public authority dependent on them. A genuine risk can produce a response that strengthens the institutions responsible for producing the risk in the first place.
There is a related problem with the language of losing control because it can blur where responsibility still resides. We increasingly say that AI decided, refused, deceived, or recommended. Sometimes that is harmless shorthand. But an AI system does not decide that it will screen job applicants, review insurance claims, or advise government officials. Institutions decide to use it. They determine what information it can reach, what tools it can use, what weight its output receives, and what happens when the output is wrong.
The discussion becomes especially imprecise when unpredictability is treated as evidence that control has disappeared. A developer may be unable to predict every response a model will generate. That is a genuine technical problem, but predictability and control are not the same thing. Human beings still decide whether the model is connected to external systems, whether it can take consequential actions, what safeguards surround it, and whether its use continues after failures occur.
Those controls themselves can fail, and ownership does not guarantee containment. But failure makes the decision to deploy an unpredictable system more consequential; it does not make the people who authorized that deployment irrelevant.
Consider what happens when we say that “the AI” rejected an applicant or generated a consequential recommendation. The description directs attention toward the machine, while the questions necessary for accountability point elsewhere: who authorized the system, what evidence justified relying on it, what limits were established, whether an affected person could challenge the result, and who had the authority to suspend its use.
These are institutional questions. They remain institutional questions even when the underlying technology becomes difficult to understand or predict.
AI can produce enormous harm without consciousness, intention, or independent purpose. It can amplify errors, enable surveillance, accelerate fraud, and operate at a scale no individual human could match. Those risks justify serious governance. But the purpose of governance should be to make responsibility easier to locate, not harder.
Humans may eventually lose meaningful control of advanced AI. If that possibility is real, we should prepare for it. What we should not do is allow predictions about future machine power to obscure present human authority.
The business value of losing control lies partly in what the idea can accomplish before control has actually been lost. It can elevate the perceived importance of the technology, make its largest developers more necessary to government and turn technical expertise into institutional power. None of that makes the warnings false. It does mean that we should pay as much attention to the human arrangements forming around AI as we do to what the technology itself may become.
Before accepting that AI has become the principal actor, we should keep looking at the people and institutions that still decide where it is used, what it is permitted to do, and who answers when those decisions go wrong.
Sebastian Saviano is an author.
