Anthropic CEO sets out three-step plan to slow frontier AI development
Embedded third-party evaluators are the first step, and the proposal shows how AI oversight is shifting from published statements to physical access.
Anthropic chief executive Dario Amodei has published an essay titled "We Must Pace the Frontier" that sets out a three-step plan for slowing the development of frontier AI models, according to reports by TechCrunch on 12 September and iTnews on 13 September 2026.
The first step places third-party evaluators, such as the non-profit METR, inside AI companies. TechCrunch reported that these evaluators would receive company badges, desks and laptops, with access described as mostly comparable to what internal risk assessment teams have. Anthropic is unilaterally committing to this arrangement. The second step calls for leading AI companies in democratic countries to agree on shared safety benchmarks and limits on the pace of development, with the United States government acting as a mediator to address antitrust concerns. The third step proposes international coordination with allied governments, along with attempted coordination with authoritarian governments, to prevent applications such as biological weapons development.
OpenAI chief executive Sam Altman responded that OpenAI will adopt the embedded evaluator approach and said pacing had been a primary topic of discussion at the company in recent weeks. TechCrunch reported that Amodei cited the OpenAI and Hugging Face security breach and the acceleration of AI capabilities, particularly self-improving systems, as reasons for acting now.
iTnews reported the essay followed an Anthropic threat intelligence report describing misuse of its Claude models, and quoted Amodei warning that swarms of AI agents could threaten the wider internet within six to twelve months.
Why it matters
The proposal moves AI governance from voluntary statements towards physical access, with outside reviewers sitting inside the building rather than reading reports afterwards. It also acknowledges that competitors cannot easily coordinate on pace without a government intermediary. The speed of the OpenAI response suggests embedded evaluation could become an industry norm quickly. Whether the second and third steps materialise will depend on institutions that do not yet exist.
The proposal moves AI governance from voluntary statements towards physical access, with outside reviewers sitting inside the building rather than reading reports afterwards.
International coordination
Allied governments, plus attempted coordination with authoritarian states, on uses such as bioweapons.
Cross-company standards
Leading labs in democratic countries agree benchmarks and pace limits, with US government mediation on antitrust.
Embedded third-party evaluators
Groups such as METR get badges, desks, laptops and access comparable to internal risk teams.
Company-internal risk teams
The existing baseline: each lab's own safety and risk assessment staff.
What you can learn from this
- Third-party evaluation is a distinct discipline from internal testing. Organisations such as METR run capability evaluations: structured tasks that measure whether a model can, for example, complete multi-step software engineering work or assist with hazardous research. Internal teams run similar tests, but an outside evaluator has no incentive to ship and reports to a different audience. Access matters because many risks only show up with the actual model weights, internal tools and pre-release checkpoints rather than a public API.
- Pre-deployment testing and continuous access answer different questions. A pre-deployment evaluation is a snapshot: the model is tested once, shortly before release, against a fixed suite. Embedded evaluators can observe training runs, intermediate checkpoints and the pace at which capabilities change between them. That continuous view is what you need if the concern is the rate of progress rather than any single release. Snapshot tests also age quickly, because post-training changes and tool integrations can shift what a model can do after the evaluation date.
- Coordination between competitors runs into antitrust law by design. Competition law in most jurisdictions treats agreements between rivals to limit output or slow product development as potentially collusive, regardless of motive. This is why the plan asks for a government mediator: a sanctioned process can give companies a legal safe harbour to agree standards they could not agree privately. The same tension appears in other safety-critical industries where regulators convene standards bodies.
- Recursive self-improvement is a feedback loop, and loops compound. The concern is a model that meaningfully accelerates the research that produces its successor, so that each generation arrives sooner than the last. Even a modest speed-up per cycle compounds over several cycles, which is why the pace of the loop is treated as a risk factor separate from the capability of any one model. Evaluations that measure AI research assistance are an attempt to see this loop forming early. The same logic is why some evaluators track how much of a lab's own engineering work is being done by its models.
- Dual-use capability evaluations try to measure uplift, not knowledge. The question in a biosecurity evaluation is not whether a model knows facts that appear in textbooks, but whether it gives a non-expert meaningful help beyond what a search engine would. That is measured by comparing task performance of people with and without the model. The distinction matters because knowledge is widely available while operational uplift is what changes the risk.
We teach this
How to use this in practice
- Read the essay and reduce it to one page. Write the three steps as three rows in a table with columns for who acts, what they give up, who checks compliance, and what already exists today. Fill every cell from the source text and mark any cell where the essay is silent. Done looks like a table where the gaps are as visible as the commitments, which is the starting point for judging any governance proposal.
- Build a twenty-prompt evaluation harness for a model you use. Pick a narrow task, such as extracting fields from invoices or writing SQL from plain English, and write twenty test cases with expected outputs. Script a loop that sends each prompt to an API or a local model, saves the response, and scores it against your expected answer. Keep the prompts and expected outputs in version control so changes to the test set are as visible as changes to the model. Done looks like a CSV with a pass or fail per case and a single accuracy number you can rerun when the model version changes.
- Review the system card of a model you rely on. Find the published model or system card and list which evaluations are reported, who ran them, and whether any were performed by an external organisation. Note which risk categories, such as cyber, bio or autonomy, have quantitative results and which are described only in prose. If no external evaluation is mentioned, record that as a finding rather than assuming none occurred. Done looks like a short checklist you can compare against the next card the vendor publishes.
- Draw the four layers of oversight described in the plan. Stack company-internal risk teams, embedded external evaluators, cross-company standards with government mediation, and international coordination as horizontal layers. Beside each, write what information flows up and what constraints flow down. Done looks like a diagram that makes clear which layers exist today and which are still proposals.
Sources
- Anthropic CEO outlines plan to slow AI development — TechCrunch
- Anthropic CEO urges AI companies to slow model development — iTnews
Our reporting is an original summary; full coverage is at the links above.
Don't just read about it — build it.
Square 1 teaches the skills behind the headlines, with every line of your work graded by AI. Find your starting point in 3 minutes.
Get your free skill report