One agent converts a large Android app, passes most requirements, and misses a dependency graph that breaks launch. Another produces less code but leaves the app usable. A binary score hides the difference, while a marketing demo hides both.
The new evaluation measures long-horizon Android work across app creation, migrations, new features, and cross-platform conversions. Five independent runs per task matter because a coding agent can finish once and fail four times under the same contract. Its leaderboard can narrow a shortlist, but your own repositories still decide whether an agent belongs in the delivery path.
Start an Android Bench 2.0 comparison with task shape
Google’s methodology describes nine app-creation tasks, thirteen migrations, six feature tasks, and two full conversions. Typical changes span hundreds or thousands of lines across many files, with migrations reaching 294 files.
That mix matters. A team planning Kotlin migrations should not average its decision with a greenfield food-delivery build. Filter the leaderboard by the work you expect, then inspect per-task completion, pass rate, and cost.
Build a workload slice
List the next ten tasks your team might delegate. Mark each as localized fix, feature, migration, creation, or conversion. Compare a benchmark only against the matching slice.
Read pass rate and completion rate together
The release announcement explains why binary grading breaks down on multi-day work. An agent can refactor forty screens and fail one edge-case assertion; pass rate records the missed contract, while completion rate preserves the useful progress.
Neither metric deserves isolation. A high completion rate with low pass rate can describe an agent that moves quickly and leaves expensive repair work. A lower completion rate with cleaner boundaries may suit a supervised exploration lane.
Use four fields for every model card
- Pass rate: how often the entire task contract survived.
- Completion: how much verified work remained useful.
- Cost: average spend per run and task.
- Variance: the spread across five independent attempts.
Ask what the harness contributes
The published setup pairs some models with agents from the same provider. Google says harness choices such as prompt caching and compact tool windows can change token use and outcomes. A model name therefore does not identify the whole evaluated system.
Record the model, agent, tool permissions, context policy, retry rules, and environment. If your organization uses another harness, treat the score as evidence about a nearby system rather than a promise about yours.
Look for runtime proof, not code volume
The benchmark runs containerized Android Virtual Devices and uses instrumentation assertions, database inspection, system-boundary monitoring, and regression suites. Its multimodal verifier walks the UI and compares screenshots with accessibility hierarchies because raw pixel diffs can punish harmless status-bar and font differences.
The verifier walks the app, captures bitmaps and accessibility hierarchies, and checks behavior without prescribing one internal architecture. That design rewards working product behavior while leaving teams free to choose implementation details.
Account for contamination and private tasks
Google keeps the long-horizon taskset private and uses an internal app plus migrations that do not exist upstream. That choice reduces memorized solutions. It also means buyers cannot inspect every task or reproduce the complete leaderboard from public files.
The public community dataset gives teams a channel for shared tasks, but it does not erase the transparency limit. Treat private-task results as vendor evidence and pair them with a small internal evaluation.
Run a ten-task local acceptance lane
Choose two fixes, two features, three migrations, one greenfield screen, and two difficult regressions from your own history. Freeze each repository state, write observable acceptance checks, and let every candidate agent attempt each task more than once.
Measure accepted completion, reviewer repair time, regressions, token and compute cost, wall time, and the number of unsafe or unexplained changes. A cheap run that consumes a senior review day is not cheap.
Turn the trial into a procurement rule
Before anyone sees a leaderboard, name the minimum pass rate for the task class, the maximum repair time, and the actions an agent may take without review. Separate exploration, pull-request drafting, and autonomous merge because each lane carries a different failure cost.
Keep the losing runs, not only the accepted patch. Tag failures as misunderstanding, environment setup, incomplete behavior, regression, unsafe change, or reviewer disagreement, then compare those categories across models and harnesses.
A candidate with a lower headline score can still be the safer purchase if its misses are visible, reversible, and cheap to repair. Repeat the slice when the model, harness, tool permissions, or repository baseline changes materially.
Our Agents API checklist covers harness boundaries. The workload-routing guide turns cost and risk into a routing rule, while the misalignment reporting guide covers abnormal traces. Visit the AI Tech hub for adjacent evidence.
