Existing methods to evaluate the performance of generative AI models are increasingly considered flawed and poor indicators for actual real-world performance. However, that does not stop AI companies from prominently highlighting benchmark results when they announce new models in press releases or company blog posts. The benchmarks AI companies choose to highlight in these announcements represent a carefully curated selection. Despite their flaws as scientific measurements, they reveal the marketing priorities of these companies. Understanding those priorities and what AI companies implicitly optimize for is an important prerequisite for understanding how genAI influences the work of journalists and how journalists can report on AI companies.

We collected 231 benchmarks highlighted across 138 model release announcements published in 2025 by 11 major AI model builders, including Google, OpenAI, Anthropic, Alibaba, DeepSeek, and Mistral. For each benchmark, we collected additional information, e.g. the affiliations of benchmark authors. In addition, we developed our own taxonomy of benchmarks based on what benchmark authors claim to evaluate. Our dataset (https://codeberg.org/cnb/benchmarking-cultures-25) and an interactive web application (https://bench-cultures.net) to explore the data are published as open resources.

Our analysis reveals, first, that the way AI companies highlight benchmarks enables little to no cross-model comparability: 63% of benchmarks are highlighted by only a single model builder. Second, benchmark development is increasingly dominated by industry-affiliated authors, raising questions about the independence of evaluation. Third, and most strikingly, some of the most popular benchmarks among model builders are shaped by "Artificial General Intelligence" (AGI) discourses. Among the five most popular general knowledge benchmarks, four are essentially AGI benchmarks referencing vaguely defined goals like reaching "expert-level AI" or creating "the final academic benchmark," without clearly defining what they measure. Given what the AGI discourse is about, we suggest AI companies prominently highlighting results on AGI benchmarks try to indicate a model’s increased potential to replace human labor.

These findings provide empirical evidence that shows the extent to which AI companies use benchmarks as narrative devices to tell stories of constant progress. For data journalists and researchers using generative AI, understanding these dynamics is an essential first step towards assessing its impact on their work. We provide Benchmarking-Cultures-25 not only to help journalists understand the tools they rely on but also to inform their own reporting on these technologies and the interests driving them.