Anthropic, now valued at $965 billion, has urged a global pause on advanced artificial intelligence development to allow society and regulators time to manage potential risks. The company warned that AI systems might soon achieve recursive self-improvement—capable of upgrading themselves without human oversight—raising concerns about uncontrolled consequences.
In a detailed blog post, Anthropic’s leadership stressed that slowing AI progress could help align technological advances with societal structures, preventing possible misuse ranging from mass surveillance to tailored manipulation on an unprecedented scale. The call urged top AI laboratories worldwide to consider formal agreements around slowing development and implementing verification mechanisms.
Despite these warnings, critics have questioned the sincerity of Anthropic’s proposal, suggesting it may be a strategic maneuver to hinder competition. The firm recently surpassed OpenAI to become the most valuable AI company, fueling skepticism about its motivations. OpenAI's CEO previously described Anthropic’s safety warnings as “fear-based marketing,” comparing them to selling expensive bomb shelters after threatening disaster.
Anthropic’s CEO Dario Amodei has been vocal about the risks of AI for some time, even clashing with the U.S. government over military uses of its technology. The company’s concern highlights the ongoing tensions between innovation speed and safety oversight in the AI industry, especially as breakthroughs promise to transform knowledge work and public services.