Artificial intelligence is being sold as one of the most transformative technologies in human history. It could accelerate scientific discovery, reshape medicine, automate difficult work and create entirely new industries. But a remarkable contradiction is now becoming impossible to ignore: some of the people closest to the technology are warning that future AI systems could pose catastrophic risks — even as the companies building them continue racing ahead.
That contradiction moved from an abstract debate into mainstream politics this week after Anthropic researcher Jacob Coxon resigned and publicly argued that leading AI companies were moving too quickly toward increasingly autonomous and self-improving systems. Anthropic alignment researcher Evan Hubinger subsequently said he personally puts the chance of AI killing all humans at greater than 10% within the next decade. That is one researcher’s estimate, not a scientific consensus or a prediction that such an outcome will occur. But the fact that people working inside a leading AI laboratory are willing to discuss risks in those terms raises a difficult public-policy question.
If the people building frontier AI believe there is even a meaningful possibility of catastrophic loss of control, should private companies alone be allowed to decide how quickly the technology advances?
The warning is no longer confined to AI critics
Concerns about artificial intelligence destroying humanity have existed for years, and many researchers reject the most extreme forecasts. What makes the latest controversy different is where the warnings are coming from.
Reuters reported on September 10 that U.S. lawmakers from both parties were calling for stronger AI rules after the Anthropic researchers’ warnings. The debate includes proposals for safety testing, independent audits and government scrutiny of the most powerful systems before or around deployment.
Anthropic itself argues that advanced AI should not be governed by industry alone. On its public policy page, the company says governments should establish rules for catastrophic-risk evaluations, independent testing and disclosure of serious safety incidents.
That creates an unusual situation. The companies developing the technology are simultaneously competing to build more capable systems and asking governments to create stronger guardrails around those systems.
Why don’t the companies simply slow down?
The obvious response is: if the risk is genuinely serious, stop building increasingly powerful models until the safety problem is better understood.
In practice, the incentives point in the opposite direction. AI has become a competition for capital, talent, computing infrastructure, customers and geopolitical advantage. A company that voluntarily slows development may believe a competitor will simply move faster. Governments face a similar dilemma: policymakers may worry about dangerous capabilities while also fearing that stricter domestic rules could allow another country to take the technological lead.
This is essentially a coordination problem. Every major player may benefit from common safety standards, but an individual player can pay a competitive price for acting alone.
That is one reason voluntary promises have limits. A safety policy adopted by one laboratory cannot automatically constrain another company, an open-source project or a state-backed program operating elsewhere in the world.
There are already risks that do not require superintelligence
The debate can become distorted when every discussion is reduced to human extinction. There are more immediate risks that deserve attention regardless of whether a superintelligent system ever escapes human control.
The Associated Press reported Thursday that Anthropic said it had blocked attempts to misuse its models for activities involving cyberattacks, surveillance and potentially dangerous biological research. Anthropic said the cases were unusual rather than typical use, and that it used the incidents to strengthen its safeguards.
Those examples matter because they show that AI safety is not only a philosophical argument about a hypothetical machine becoming smarter than humanity. The technology can amplify existing human capabilities — including harmful ones — long before anything resembling science-fiction superintelligence arrives.
But extinction claims should not be treated as established fact
There is an equally important caution on the other side of the argument. A researcher’s probability estimate is not empirical proof that AI has a 10%, 20% or any other measurable chance of destroying humanity. There is no controlled experiment that can establish a reliable probability for an unprecedented event of that kind.
Experts disagree sharply about how quickly artificial general intelligence might arrive, whether today’s machine-learning approaches can produce superintelligence, how controllable future systems will be and whether existential-risk scenarios are receiving too much attention compared with present-day problems such as fraud, misinformation, surveillance, bias, labor disruption and concentrated corporate power.
A responsible debate therefore has to resist two temptations at once: dismissing serious warnings simply because they sound extreme, and presenting speculative worst-case scenarios as if they are inevitable.
Washington is beginning to react
The political response is becoming harder to ignore. Reuters reported bipartisan interest in new safeguards, including independent security audits for powerful AI models. Separately, Senator Bernie Sanders and Representative Greg Casar announced plans for legislation that would seek to ban artificial superintelligence and temporarily pause certain advanced AI development until federal safety rules exist. Their proposal is politically ambitious and far from guaranteed to become law, but it illustrates how quickly the issue is moving from research circles into mainstream politics.
The difficult part will be defining what deserves regulation. Rules that are too weak may become little more than paperwork. Rules that are too broad could suppress useful research, protect incumbent technology companies from smaller competitors or push development into less transparent jurisdictions.
The deeper issue is who gets to choose the risk
The most important question may not be whether one particular AI doomsday forecast is correct. It is whether society has built institutions capable of making decisions when a powerful new technology carries uncertain but potentially enormous consequences.
When a pharmaceutical company develops a new medicine, it cannot simply decide for itself that the drug is safe enough and release it without external rules. Airlines do not individually determine what level of aircraft failure is acceptable. Nuclear facilities operate under extensive oversight precisely because low-probability failures can have extraordinary consequences.
Frontier AI is different in many ways, but the underlying principle is relevant: when the possible downside extends far beyond the company taking the risk, the public has a legitimate interest in how that risk is managed.
Progress and precaution do not have to be enemies
The choice does not have to be between stopping AI completely and allowing an unrestricted technological race. Governments could require standardized evaluations for the most capable models, independent audits, mandatory reporting of serious safety incidents, stronger cybersecurity around model weights and computing infrastructure, and clear thresholds that trigger additional scrutiny as capabilities increase.
International coordination will also matter. A safety regime confined to one country becomes less effective if the most advanced systems can simply be developed elsewhere.
The challenge is doing this without freezing innovation or turning regulation into a barrier that only the richest technology companies can afford to navigate.
A question the AI industry cannot avoid
Artificial intelligence may ultimately deliver benefits that justify enormous investment and rapid development. The darkest predictions may prove wrong. Future systems may remain controllable, and technical safeguards may improve faster than critics expect.
But uncertainty cuts both ways.
If leading researchers are publicly saying that catastrophic outcomes deserve serious consideration, then society should not require certainty of disaster before demanding serious safeguards. At the same time, extraordinary claims require scrutiny, evidence and open disagreement rather than fear-driven policymaking.
The controversy exposes the central paradox of the AI era: humanity is racing to build machines more capable than anything it has created before, while still debating whether it knows how to control what comes next.
The question is not simply whether we should build powerful AI. It is who gets to decide when the benefits are worth the risks — and what happens if the people making that decision are also competing to win the race.
Sources
Reuters — U.S. lawmakers call for new AI rules after Anthropic researchers’ safety warnings
Associated Press — Anthropic says it blocked misuse of its AI
Anthropic — AI policy and advanced AI framework
U.S. Senator Bernie Sanders — proposed artificial superintelligence legislation
Top New Trends reports on fast-moving developments using publicly available reporting and primary sources. Claims about future AI capabilities and catastrophic risk remain disputed and should be understood as forecasts, not established outcomes.
