A new report published Thursday by the AI safety research organization METR reveals that Anthropic is using AI to build more powerful AI, with models improving roughly twice as quickly as they were a year ago. The findings, which come after an 11-week investigation into the company's research and development, suggest that benchmark scores indicate “that Anthropic has made roughly two years of historical progress over the last year.”
METR researchers stated they are confident this acceleration is not simply due to increased computing power. Instead, they reported that the majority of the acceleration as of October is due to AI, with its contribution growing over time. METR wrote that employees using Anthropic’s most capable internal model on average self-reported being around 12 times as productive at their respective work as they would have been without AI assistance, around double what staff had reported just a few months earlier while using Mythos 5.
While much of the current acceleration appears to come from AI assisting human researchers, METR warned that “there are some signs that fully autonomous contributions will play a larger role soon.” The researchers found no clear signs of diminishing returns or observable bottlenecks that could prevent “even faster acceleration.”
The report identified several unreleased models used by Anthropic researchers, including codenames Model 2, Model 3, and Model 4, which completed training in July, August-September, and September-October, respectively. METR also identified an unreleased model dating back to 2022, called Unreleased 2022. METR noted that these are the codenames shared with them and that the company has different internal codenames.
Anthropic provided METR with non-public information, including confidential research and model performance data, which METR said surpassed their initial expectations. However, the company did not grant unrestricted access. Anthropic also exercised its right to redact a majority of the evidence METR found most informative for the public version of the report.
