Google has overhauled its Android Bench, the popular evaluation suite for AI coding assistants that specialize in Android app development. The benchmark now runs on the standardized Harbor framework, replacing the previous mini-swe-agent v1 setup, and an updated leaderboard puts Anthropic’s Claude 3.5 Sonnet at the top.

What actually changed in Android Bench

Android Bench is a set of real-world Android coding tasks that gauges how well large language models handle common developer challenges—generating layouts, wiring up UI logic, fixing bugs, and implementing features. Until now, those tasks were orchestrated through mini-swe-agent v1, a lightweight harness that presented models with a Git-based development environment.

With this update, Google has migrated the entire benchmark to Harbor, an open-source evaluation framework originally incubated by the AI community to bring consistency to agentic coding benchmarks. The move is not cosmetic: Harbor defines each task as a self-contained Docker environment with a fixed prompt, a canonical solution, and a rigorous scoring protocol. Every model that takes the test now faces the exact same challenge in the exact same sandbox, eliminating the variability that older harnesses sometimes introduced.

Harbor also allows Android Bench to test more than just code generation. It measures a model’s ability to navigate the Android project structure, run build tools like Gradle, and even install the resulting APK in headless emulators to verify the app actually runs. Tasks are graded on pass/fail criteria that check both compile success and the correct implementation of the specified feature.

The refreshed leaderboard reflects these expanded tests. Claude 3.5 Sonnet from Anthropic—referred to in some early reports as “Claude Fable 5,” likely an internal code name—claims the highest overall accuracy, followed by a mix of commercial and open-weight models. The exact scores and rankings are now public on the official Android Bench site.

Beyond the model shuffle, the platform now invites third-party contributions more easily. Because tasks are packaged as Docker images, any researcher can submit a new challenge or propose a fairer scoring rubric, making the benchmark a community-driven yardstick rather than a black box controlled by a single team.

What the update means for you

For everyday Windows users

You might never touch Android Bench directly, but the results trickle down to the tools you use. If you’re a Windows user who occasionally runs Android apps via the Windows Subsystem for Android or builds cross-platform projects, an AI coding assistant that ranks high on Android Bench is more likely to give you correct Kotlin or Java snippets, generate functional XML layouts, and debug Gradle build errors on the first try. As model makers compete for the top spot, the quality of AI-powered code completion in editors like Visual Studio Code—which runs on Windows—should improve for Android-related work.

For professional developers

If you write Android apps for a living, the new Harbor-backed Android Bench gives you a transparent, apples-to-apples comparison of AI coding assistants. Instead of guessing which model will work best in your CI/CD pipeline, you can look at the published pass rates for exactly the kind of task you face—adding a RecyclerView, handling permissions, or integrating Room database. The shift to containerized tasks also means that if you have an in-house model, you can run the identical evaluation privately and see how it stacks up against the public leaderboard. That alone makes Android Bench a practical procurement tool for teams considering an AI copilot license.

For IT professionals and platform architects

The modernization to Harbor matters for governance and repeatability. Harbor’s open-source, container-centric design lets your team audit the test methodology, fork the benchmark, and even add proprietary tasks that reflect your internal coding standards. Because the environment is deterministic, compliance teams can verify that a model’s performance isn’t cherry-picked. If you’re evaluating AI tools for enterprise deployment, the new Android Bench provides a defendable, vendor-neutral scorecard.

How we got here

Android Bench debuted in late 2023 as part of Google’s effort to measure AI progress on mobile development—a domain that mainstream coding benchmarks like HumanEval and SWE-bench often neglect. The original harness, mini-swe-agent v1, borrowed from the approach used in SWE-bench but was tailored for the Android SDK. Models were given a GitHub repository with a failing test or a feature request, and they had to modify source files and push a pull request.

While that setup worked for early comparisons, it introduced headaches. Subtle differences in how agents set up the development environment, resolved dependencies, or interacted with the Android SDK led to run-to-run variance. Two models could get the same prompt but face slightly different builds, making it hard to declare a true winner. The community called for a more rigorous standard, and Google listened.

Harbor emerged as the likely successor because it had already proven itself in other agentic benchmarks. Developed by a consortium of academic and industry researchers, Harbor treats each task as an ephemeral Docker container with a fixed OS image, locked toolchain, and a single evaluation path. The framework had been battle-tested on general-purpose software engineering tasks, but Android Bench is its first high-profile mobile application.

The mid-2025 rollout also coincides with Anthropic’s release of Claude 3.5 Sonnet, a model that has topped several coding leaderboards by pairing strong reasoning with a large context window. Google hasn’t confirmed whether it contributed engineering resources to help migrate Android Bench to Harbor, but the change aligns with the company’s broader push toward open AI evaluation standards.

What to do now

If you’re an Android developer
Visit the updated Android Bench leaderboard and filter tasks by tags relevant to your work (e.g., “UI,” “networking,” “persistence”). Note the models that rank highly in your areas and consider running a small head-to-head trial on a private branch of your app. Most of the top models offer free API tiers or pay-as-you-go access, so a two-hour experiment can tell you more than any public chart.

If you manage a dev tools budget
Ask your vendor how their AI coding assistant performed on the Harbor-based Android Bench. Demand granular results, not just an aggregate score. The open-source nature of Harbor means there’s no excuse for a black-box claim—providers should be able to point you to a reproducible run on your laptop or CI server.

If you’re evaluating a custom model
Clone the Android Bench repository (now probably under https://github.com/google-research/android-bench) and spin up the Docker images. Harbor’s documentation will walk you through hooking your model’s API into the evaluation loop. Run the full suite and compare your numbers against the public leaderboard to identify blind spots.

If you’re a researcher
The Docker-based task packaging makes it trivial to submit new tasks. If you’ve encountered a common Android programming pitfall that isn’t represented, create a task definition and issue a pull request. Google has signaled that it welcomes community contributions and will maintain a verified task certification process.

What to watch next

Expect the Android Bench leaderboard to become more volatile. With a level playing field now in place, model builders will tune their systems specifically for the benchmark’s pass rates—a bit like how CPU makers optimize for SPEC. Open-weight models such as Code Llama and StarCoder are already catching up, and Google’s own Gemini series has not yet been fully optimized for the new framework. A rematch is almost certain before the end of the year.

The bigger story is the migration of mobile-AI evaluation away from ad-hoc harnesses and toward containerized, community-governed platforms. If Harbor succeeds here, it could become the de facto standard for benchmarking AI coding assistants on any platform, from iOS to embedded systems. For anyone who writes code for a living, a world where models compete on transparent, reproducible tests is a world where you can finally trust the leaderboard.