Anvra

The problem

Product teams collect feedback constantly: support tickets, session replays, analytics anomalies, interviews. Almost none of it turns into a decision. The signal exists. The path from signal to shipped work is manual and lossy.

What I built

A pipeline where separate agents own separate stages. Raw signals are ingested and held immutably. One agent clusters them into opportunities. Another produces a decision with explicit reasoning. Another writes a PRD in a fixed set of sections. Another decomposes it into engineering tasks.

Stages don't call each other directly. Each writes a record when it completes, and a worker picks that up and decides what runs next. If the process dies mid-pipeline the work is still in the database rather than lost.

Nothing an agent produces is saved automatically. It proposes, the proposal is validated against a schema, and only then is it written.

Measuring whether the agents actually work

This is the part I care most about. Model output that can't be measured shouldn't be shipped.

Every prompt is a versioned pack: system instructions, context assembly, and output contract bundled together and labeled. No version reaches production without an evaluation run against a locked baseline.

Scoring is rubric-based with an LLM-as-judge, and the judge is calibrated against known-good and known-bad outputs before every baseline run. An uncalibrated grader quietly corrupts every result downstream.

I benchmarked three frontier models against a hard-scenario set: 86.1 with a 100% pass rate, 85.1 at 93.8%, and 81.9 at 87.5%. I routed production traffic to the model that scored slightly lower, because its failures were concentrated in rubric dimensions that mattered less for the task and its cost and latency were better. That decision came from reading the evidence, not from a threshold.

Testing non-deterministic systems

The end-to-end suite asserts on schema conformance, state transitions, and persisted database records rather than on generated text. The machine either worked or it didn't, and that's a deterministic question. Whether the output was good is a separate question the evaluation layer answers.

Six pipeline scenarios across 140 seeded signals, plus roughly 1,500 unit and integration tests on the orchestration layer.

Decisions worth explaining

Append-only signals with database-level constraints. An invariant enforced in application code is a suggestion. Enforced in the database, it's a rule.

Propose-then-confirm. No agent writes directly to the database.

Cost as a release signal. Requests route by complexity to the cheapest model that clears the quality bar, with escalation when confidence drops.

Outcome

Deployed and running. The evaluation harness is the piece I'd carry into any team building with agents. It's the difference between shipping model output and knowing whether it's good enough to ship.