We’re really excited about the adoption of ProgramBench (Anthropic, GLM, Kimi). Looking forward to supporting investigations of long-horizon coding at scale. Expect more models, more harnesses,…

This and more explained in the blog:

oh gosh, this is kind of a big deal DO NOT ASK YOUR AGENTS TO DO TDD! i now have empirical evidence that Test Driven Development is harmful for coding agents what other popular skills do you want me…

下班前，ProgramBench。给 AI 一个编译好的二进制和文档，要求从零重建出行为一致的代码库。终于有基准不是玩套壳刷榜，而是逼模型真正理解程序逻辑。很多所谓的代码生成其实背题库，这个直接看成果能不能对上二进制行为，有点意思。

Here's the link to the system card, this is on p.212

More on metrics:

mini-swe-agent has been powering all of our most recent projects. It's codebase is extremely compact (the main agent class is just 100 lines), while having excellent performance on SWE-bench & broad…

The dominance of GPT 5.5 xhigh is apparent in almost every instance.

ProgramBench contains 200 tasks, covering projects from ripgrep and jq to FFmpeg, SQLite, and PHP. Its primary metric is the number of fully solved instances. Claude Opus 4.7 scored 3.0% on the…