On July 31, Supabase released an open-source evaluation framework that doesn’t just test whether an AI coding agent can write code — it measures whether that agent can safely build, deploy, and fix a real database-backed application. The first published results show even top assistants still make security and workflow mistakes that could quietly break production systems. The project, called Supabase Evals, runs under the Apache-2.0 license and was first highlighted by MarkTechPost on August 1.
A Benchmark Built for Real Work, Not Toy Tasks
Supabase Evals divides its scenarios across three axes to mirror what developers actually do: product areas, technical topics, and work stages. The product list includes database, authentication, storage, Edge Functions, Realtime, cron, queues, vectors, and the Data API. Ten technical topics span Row Level Security (RLS), SQL, migrations, SDKs, observability, self-hosting, tests, and declarative schemas. The stages — build, deploy, investigate, resolve — ensure an agent is tested on everything from creating a new schema to diagnosing a failing Edge Function or repairing a broken RLS policy.
The scenarios themselves come from support tickets, bug reports, and GitHub issues. One task might ask an agent to construct an application schema; another might supply a broken hosted project and require investigation; a third might demand a repair to RLS behavior. This operational realism is the framework’s sharpest edge. It catches failures that a simple code-generation benchmark would miss.
The public suite feeds a leaderboard and captures representative tasks. A separate, deeper regression suite tracks known failure modes and refreshes daily, but its results stay internal. Separating the two is a sensible move: it lets Supabase test a newly reported bug without turning one narrow issue into a permanent distortion of a model’s public ranking.
How the Testing Gets Real
The technical design is the release’s central contribution. Each scenario runs against a real environment, with a hosted-like project surface and a local CLI project created in containers. Agents invoke the actual Supabase MCP server or the actual Supabase CLI — not a mock interface. For hosted-style tasks, a component called platform-lite exposes a Management API-compatible interface backed by @supabase/lite. This means tools like the Supabase MCP server work against a lightweight but compatible target. The test can seed database state, deployed Edge Functions, and observability logs before the agent begins.
For CLI-oriented scenarios, the framework starts a Docker sandbox that includes Bash, file operations, and the real Supabase CLI. An agent can use commands like supabase init, supabase start, supabase db, and supabase test against a local stack. After the agent finishes, the harness exports the modified workspace and checks the delta — it doesn’t reward the pre-seeded environment. That focus on the agent’s own changes adds credibility.
Scoring blends deterministic checks with an LLM-as-a-judge. Deterministic checks verify whether a user can access the correct data, a policy blocks the intended operation, an Edge Function returns the expected result, or the project’s test suite passes. The LLM judge covers requirements that can’t be reduced to a boolean — it reviews transcripts, generated files, tool calls, SQL, and real outputs. Each agent gets one retry after a failure, which can filter out transient tool errors but also means a retry-reliant agent may be fine for supervised development but risky for unattended remediation.
The Most Worrying Failures Are the Silent Ones
Supabase’s initial findings are less a model horse race and more a catalog of where modern agents still take shortcuts. The company found agents commonly hand-write database migrations even when a project already uses declarative schemas — the single source of truth for database structure. Agents writing Edge Functions often manually verify authentication through supabase-js instead of choosing @supabase/server, a newer library designed to reduce boilerplate around secure Edge Function work. Supabase has since published package-selection guidance and revised its agent skills to steer behavior.
The RLS failures are more consequential. A malformed policy can expose data across user boundaries, cause silent application errors, or spawn support incidents that are hard to reproduce. Supabase’s April Agent Skills project had already warned that AI tools frequently omit RLS policies, misuse user metadata for authorization, expose service-role credentials in front-end code, or create views that bypass RLS unless security_invoker = true is set. In a fintech, healthcare, or internal business app, a plausible-looking agent patch can pass superficial testing while quietly weakening tenant isolation. Supabase Evals doesn’t eliminate that risk, but it makes it possible to turn a bad incident pattern into a regression test — and to block a skill, documentation change, or agent configuration that reintroduces the problem.
What This Means for Your Windows Development Workflow
For Windows developers who run Docker Desktop, Node.js, and the Supabase CLI locally, the repository is deployable now via pnpm. But local-stack mode is a serious test environment, not a quick demo: it requires a working Docker daemon, API credentials for the selected model provider, and Supabase’s default ports 54321–54329 to be available. The harness mounts the host Docker socket into its sandbox so supabase start can launch sibling containers — that’s appropriate for an isolated developer workstation or a carefully controlled CI runner, but it’s not something to run casually on a shared build host.
A team maintaining a Supabase-backed application can add custom scenarios before allowing an AI agent to modify database code or operational runbooks. Imagine a test for a tenant-isolation policy, an Edge Function authentication regression, a migration conflict, or a Storage permission issue. The agent’s output gets tested against a real local stack before it reaches a pull request or a staging environment. For IT admins evaluating coding assistants, the framework provides reproducible experiments: you no longer have to guess if a model plus a specific harness configuration will respect your security requirements. The result depends on the model, the agent harness, the enabled skills, MCP availability, the CLI version, and whether the agent reads fresh documentation — Supabase Evals represents all of those explicitly.
A Snapshot of Model Performance — and Shortcomings
Supabase’s published scores offer a point-in-time glimpse. In the Build stage, Opus 5 and Kimi K3 both scored 100 percent without loaded skills. Skills narrowed the gap for other configurations: Sonnet 5 moved from 78 percent to 100 percent, GPT-5.6 Sol from 89 percent to 100 percent, and GPT-5.4 mini from 78 percent to 89 percent. But these figures are vendor-published and should be taken as directional, not permanent rankings.
The documentation behavior is especially revealing. Codex-based agents, including a GPT-5.6 configuration, read about eight documentation pages per scenario, compared with roughly two pages for Claude Code. Claude Code reportedly checked Supabase documentation in fewer than 40 percent of scenarios even when skills were loaded. That doesn’t mean one product is inherently safer — a capable model that skips current docs can confidently implement an outdated pattern, while a smaller model with a well-designed skill, tool access, and a verification workflow may be a safer fit for constrained tasks.
Getting Started with Supabase Evals on Windows
If you want to test an agent locally, start with these prerequisites: Docker Desktop running, Node.js installed, pnpm available, and API keys for your model provider (Anthropic, OpenAI, etc.). Clone the repository from GitHub, install dependencies with pnpm install, and configure your environment variables. Make sure ports 54321 through 54329 are free — the Supabase local stack grabs them all. Run the framework in an isolated development VM or a dedicated CI runner; avoid doing this on a machine that hosts sensitive containers or files.
Run the public benchmark first to get a baseline for your chosen agent. Then, consider adding internal failure cases that matter to your organization: a migration that conflicts with an existing declarative schema, an Edge Function that must use the correct auth library, or an RLS policy whose breakage would be a data exposure incident. Treat the results as a directional signal, not a pass/fail certificate. An agent that scores 100% on the public scenarios can still miss edge cases unique to your app.
The Road Ahead for AI Backend Quality
Supabase plans to expand edge-case coverage, improve scoring stability, and eventually promote mature regression scenarios into the published benchmark. The company is also working on a CLI command and MCP tool that would let agents report where they struggled — that could close the feedback loop between evaluation and model improvement. The real test will be whether teams outside Supabase turn the open framework into their own deployment gates, particularly for the unglamorous failures that only become visible after an AI-written backend reaches real users. As coding assistants become more embedded in backend workflows, frameworks like Supabase Evals may become as routine as unit tests.