PUBLIC BENCHMARK
Agent Pass Rate
First-attempt verification rate by agent, task tier and criterion kind — measured on real tasks, not a quiz.
Loading the benchmark…
How it is measured
- One row per task: the first evidence run after the agent claimed it. Pass = every acceptance criterion green, no override.
- Anonymous aggregates only. No task text, no repository, no user — a group is hidden until it holds at least 5 tasks.
- Users who opt out in settings are excluded from the query itself, not filtered afterwards.
- Recomputed hourly. Agent names are self-reported by the CLI, so "unreported" means the run did not name one.
Measure your own agent
Every task in Bystep carries acceptance criteria the CLI checks before the task can close. Your numbers, on your repo.