Managers were spending ten-plus hours a day grading field work — shallowly, incompletely, and never fully without bias. Now an AI rates every task overnight, with the reason attached, and it had to prove itself against a thousand human judgments before it was allowed to.
Orders end at a customer's front door, and the brand's reputation is decided in those visits. Judging them was arithmetic that couldn't work: 25 technicians × 5 tasks × 5 minutes of honest review is 10+ hours a day, every day. A manager could be complete or deep, never both — and an in-charge rating their own team is never fully unbiased. So ratings were skimmed, sampled, and quietly distrusted.
This is a perfect AI job, for two reasons. The volume is high and the evidence trail is complete: every task already carries transcribed customer calls, images and selfies from the site, logged actions, timestamps, the technician's designation, and the customer's own rating. Nothing new had to be collected — the judgment just had to be done, on everything, every night.
A single overall score hides everything useful. The system scores each dimension on its own — communication, workmanship, compliance, punctuality, sentiment — and then aggregates, so a low score arrives with its reason attached.
Per-dimension scoring is the difference between a grade and a coaching plan.
The evidence lived across 10+ tables. A nightly cron pipeline runs gather → join → transcribe → score → aggregate → club, rating roughly 10,000 tasks a night on a Gemini + OpenAI + Sarvam stack. The unglamorous truth: most of the work was data engineering before any model saw a prompt.
An AI that judges people doesn't get the benefit of the doubt. Before going live it was tested against 1,000+ tickets already rated by in-charges, dimension by dimension, and shipped only past a 90%+ accuracy bar against those human judgments. In production, self-consistency works as a tripwire: task status and rating must correlate, so a broken correlation surfaces edge cases automatically — and errors alert the involved parties by email and Slack. Failures announce themselves.
Technicians can challenge a rating, and challenges don't just flip a score — they turn into new parameters and rules. The rubric is versioned by dispute: the people being judged are also, continuously, the system's toughest evaluators.
Managers got back a minimum of 10x the time they spent judging, every day — redirected to coaching. Coverage went from sampling to everything, and the judgment became consistent: the same rubric, applied the same way, to every task. The transferable lesson: AI-as-judge works when the evidence trail is complete and the rubric is explicit. Evals are an engineering problem, not a research luxury.