I added AI capabilities to a business-critical platform without ever taking it offline, led the team that did it, and it now automates work people used to do by hand.
Leadership wanted the platform to understand plain English — free-form input, text or voice, triggering real business workflows: bulk ticket updates, order actions, data lookups. The workflows were load-bearing; turning the system off was never on the table. Neither I nor my core team of six had built production event-driven systems when the mandate landed — the constraints on paper became the design inputs.
A wrong answer wasn't a bad demo. It was a wrong order.
Worked in demos, collapsed on real inputs — a monolithic prompt is a monolithic failure domain.
Correct on a whiteboard; would have taken the business offline for a quarter.
Any slow model call becomes a user-facing timeout across the whole flow.
Router (intent), retriever (RAG), planner (tool choice), executor (actions) — microservices behind the existing system. When something fails, you know which step failed, and you can test, trace and improve each independently.
Kafka for ordered event streams, RabbitMQ for retries and dead-letter queues. The user's request is acknowledged instantly; the heavy work happens in the background; the result comes back through the notify step. Slow model calls degrade gracefully; nothing blocks. The legacy platform never learned the agents existed.
Fine-tuned LLaMA and Mistral on company data — including designing the Q&A format — so the system understood our domain. RAG, multimodal across text and image, handled our data.
In a synchronous system, when something breaks you get one error, in one place, with a stack trace pointing at it. In an event-driven system, a single user request becomes many events hopping between services over time — the evidence is scattered across the whole system instead of sitting in one log. I knew debugging would get harder, and paid for it up front: three unplanned weeks building distributed tracing (follow one request across every service it touches) and a replayable event log (re-run any past sequence of events and watch exactly what happened). Nobody had budgeted those weeks. They repaid themselves the first bad night.
The first version let agents retry freely. One malformed input, one night: a retry storm that duplicated downstream actions — on a commerce platform, that's real operations executed twice. The deeper lesson: retries assume failure is transient, and a malformed input is not.
FIX, SHIPPED WITHIN A WEEK — REDIS-BACKED IDEMPOTENCY KEYS ON EVERY ACTION + A CIRCUIT BREAKER.Every integration I've designed since starts with the failure path. I consider that incident the most valuable three weeks of my career.
Ship the observability before the second agent, not after the fourth. I could see the system worked long before I could see why.