Comparison benchmark: ClawBench (live-site browser agent eval)

[reacher-z]

Hi team,

ClawBench is already referenced in your list (paper_by_key/paper_benchmark.md, paper_by_key/paper_long-horizon_tasks.md, paper_by_env/paper_web.md). Opening this as an informational / structural suggestion rather than an addition request.

Context for the project, in case useful:

• 153 everyday tasks on 144 live production websites (15 life categories)
• Unlike WebArena/VisualWebArena, runs directly on real sites via a submission-interception layer (CDP + Chrome extension) that blocks only the final write request
• 7 frontier models evaluated; best passes 33.3%
• Paper: https://arxiv.org/abs/2604.08523
• Repo: https://github.com/reacher-z/ClawBench

The paper is already indexed at paper_by_key/paper_benchmark.md. Would you consider creating a dedicated 'live-site benchmarks' subsection (ClawBench, Online-Mind2Web, and similar) to separate them from sandbox benchmarks (WebArena/VisualWebArena)? The execution model differences are substantive enough that readers looking specifically for production-site evals currently have to cross-reference multiple entries.

Happy to help assemble the subsection if useful.

Disclosure: I'm affiliated with ClawBench.

[boyugou]

Sounds reasonable. I have also read your nice and timely paper and congrats on the great work.

Generally, we group papers automatically by keywords, so that people can accurately find them in All Papers / readme. Then I feel it's indeed nice to add certain keywords for realistic-web benchmarks such as ClaudeBench, OL-M2W, AssistantBench, WebVoyager, etc.

[boyugou]

(Auto-reply from Claude Code.) Thanks for the thoughtful suggestion! Added a [realistic website] keyword tag to ClawBench, Online-Mind2Web, and AssistantBench in commit c92676a, which groups live-production-site benchmarks together and separates them from sandbox benchmarks like WebArena/VisualWebArena. Readers can now browse them via the keyword grouping. Leaving this issue open for now in case you or others have additional suggestions on the live-site taxonomy.

[boyugou]

(Auto-reply from Claude Code.) A quick follow-up — we added the [realistic website] keyword to ClawBench, Online-Mind2Web, and AssistantBench, but didn't do a deep pass over other benchmarks in the list. If you'd like to help refine the taxonomy (e.g. apply the tag to other live-site benchmarks you're aware of, or suggest additional distinguishing keywords), a PR would be very welcome!