Raw capture â raw/using-llm-as-a-judge-scoring-to-measure-your-software-factory.md
url: https://www.warp.dev/blog/using-llm-as-a-judge-scoring-to-measure-your-software-factory date_fetched: 2026-09-22
If you want to get the most out of coding agents in your organization, you need to stop guessing how well your agents are performing, and start measuring.
There are a couple of ways to do this. The typical approach is DORA metrics: PR merge rate, cycle time, defect rate, time to fix errors, etc. If those metrics are all going in the right direction as you increase agent usage, it's a good signal you are getting value from coding agents.
Thereâs a second approach though, that you should also consider: directly scoring coding agents using LLM-as-a-judge. Since agents provide a complete digital record of their work, you can examine and grade past sessions, see where they are deficient, and adjust going forward.
Scoring forms the basis of agentic self-improvement, where observer agents automatically suggest changes to improve agent ROI based on past scores of how the factory is performing.
There are a few prerequisites for setting up an effective scoring system. Iâll illustrate the primitives using the built-in scoring infrastructure in Warp Factories, but you can also create something similar on your own.
Here is the tl;dr:
Letâs take a closer look:
First, you need a record of prior agent traces that your scorers can grade. These traces should include not just the agent conversation, but the agentâs entire âinput and output;â and they should be stored in the cloud and be accessible via API so agents can analyze them.
âInputsâ are prompts, tool calls and MCP results, input images, etc. âOutputsâ should include all artifacts created by the agent like PRs, specs, screenshots, etc; anything that would be helpful in judging whether the agent did its job. In Warp Factories we automatically store all this info and make it API accessible (potentially in a companyâs own storage). Depending on your factory approach, you may have to do some infrastructure work to set this up.
Second, you need a way of defining and triggering âscoring agents.â A scoring agent takes a prior agent trace as an input and returns a grade. Each scorer typically focuses on a single dimension like cost or quality, and is defined by a prompt, classification instructions, and a judge model to use. Youâll also need a place to store and view the aggregate scores. Again, this is built into our factory infra; if you are building your own youâll want to use some sort of cron-based cloud agent to score prior runs.
You can define scoring agents along different dimensions:
For example, hereâs the definition for a custom scorer that checks for redundant test creation, a common failure mode we were seeing in our internal factory.
It contains a judging prompt (our full redundant tests scoring rubric for reference):
Along with a set of output classifications â what counts as a âpassâ â
and a sampling rate, indicating what percent of runs to score.
When a scoring agent runs, it loads an agent trace, brings all its inputs and outputs into context, and then prompts an LLM to judge the run. The output is a classification like in the above example.
You wonât necessarily want to score every run, since scoring itself costs money. Instead, youâll want to (third) decide on a sampling strategy. It could be percent-based, it could be classifier based (e.g. âscore all my front-end tasksâ), etc. For our internal factory, scoring currently accounts for about 3% of total token costs â thatâs a reasonable amount to get visibility into agent performance.
Over time, (fourth) youâll build up a corpus of your scored runs. At the simplest level, you can use these just like DORA as another measurement of the efficacy of your factory. You can graph how the metrics are changing over time, catch regressions when they get worse, etc. Depending on how your factory is set up, you can try to correlate changes to models, skills and context with improvements (and regressions).
In the above graph you can see that our scorer thinks we are mostly avoiding redundant tests, but there are a few failing runs every day. To investigate, you can click into the failures and examine what the coding agent did and also examine the scorer run itself, since itâs just another agent, to understand why it thinks these coding agent runs produced redundant tests. You may notice patterns, and then adjust the skills which drive your agents, so that they write tests more sparingly.
Once you get a feel for checking your scorers by hand, youâll probably want to (fifth) automate how they are used, and create an actual learning loop. In Warp Factories we call this âself-improvement,â and you can learn more about it here. The tl;dr is that scorers can be input into another agent loop that synthesizes their output in batch and creates updates to the factory definition automatically.
Scorers also (sixth) form the basis of more advanced optimizations like benchmarking, where you test different model configurations against your factory to optimize its cost and performance. If you want to learn about benchmarking, check out this post.
In sum, if you arenât currently scoring your coding agents, you are missing a crucial layer of visibility into how they are performing and how you might improve them. Itâs a bit of work to set up, but in an age where more and more of your companyâs software production depends on how efficiently your agents work, itâs well worth the effort to gain that visibility.
If you are interested in learning more about Warp Factories and how they are helping companies scale development on open, observable infrastructure, you can request early access here. We are offering up to $10k in usage to qualified companies.
Book a demo and weâll walk you through the workflows that map to your stack.
Sep 5, 2026Product
8 min
8 min
Sep 3, 2026Product
6 min
6 min
Aug 18, 2026Product
14 min
14 min
Aug 4, 2026Product
6 min
6 min
Aug 3, 2026Product
7 min
7 min