A brain for a system that couldn't stop
A legacy commerce platform carrying real revenue, and a mandate to make it understand plain English. Nobody was willing to pause the business to get it.
MY ROLE — TECH LEAD, HANDS ON
TEAM OF 6
10 MONTHS
THE SITUATION
Leadership wanted GenAI capability inside a platform people used every day to run their business. The workflows were load-bearing. Turning it off was never on the table.
THE CONSTRAINTS
No downtime, no rewrite, and a team that had never built event-driven systems. Free-form user input had to trigger real business actions — so a wrong answer was not a bad demo, it was a wrong order.
OPTIONS I REJECTED, AND WHY
One large prompt with tools attached
Worked in a demo, collapsed under real inputs. No way to isolate which step failed.
Rewrite the monolith around the agent
Correct on a whiteboard. Would have taken the business offline for a quarter.
Synchronous calls between services
Any slow model call became a user-facing timeout across the whole flow.
WHAT I CHOSE
Specialised agents for reasoning, retrieval and action, sitting behind the existing system as microservices. Kafka and RabbitMQ for pub/sub so nothing blocked. The legacy platform never learned the agents existed.
THE TRADEOFF I ACCEPTED
Debugging got harder. A synchronous system fails in one place; this one fails across a timeline. I paid for that up front with tracing and a replayable event log — about three weeks nobody had budgeted.
WHAT WENT WRONG
The first version let agents retry freely. A malformed input produced a retry storm that duplicated downstream actions overnight. Idempotency keys and a circuit breaker went in the following week, and that incident is why every integration I have built since starts with the failure path.
WHAT I'D DO DIFFERENTLY
Ship the observability before the second agent, not after the fourth. I could see that the system worked long before I could see why.
- zero
- downtime during rollout
- manual steps
- removed from the daily flow
- 6
- engineers onboarded to event-driven