All work

Case 01

Building an AI agent platform

What changed A single bad file can no longer take an agent down, and every provider error now names a precise reason.

Teams across the company built their AI agents on one shared platform, and I worked on it: the engine behind every turn, the studio where agents are made, and the gateways to chat and tools.

An agent answers “Summarise yesterday’s failed checkouts”, step by step: memory, a skill, tools, the model, then a language check.
My role
AI engineer on the platform team. I took fixes and features end to end: root cause, failing test, review, release, then watching it.
Scope
A multi-tenant agent platform used by teams across the company.
Stack
Go · TypeScript/React · PostgreSQL · MongoDB · Docker · Kubernetes · AWS
Shipped
Guest access across the services, a reply-language resolver, per-turn outcome records, and the fix for an outage caused by a single bad file.

When something breaks

How I fix a bug in production.

01Report

Something breaks

A team reported that their agent had stopped replying. Before touching any code, I went to the logs.

service:engine status:errorlogs
⌕service:engine status:error
14:02:11ERRORengineturn failed agent=support-agent parse frontmatter: yaml: line 2
14:02:40ERRORengineturn failed agent=support-agent parse frontmatter: yaml: line 2
14:03:05ERRORengineturn failed agent=support-agent parse frontmatter: yaml: line 2
one agent · every turn · all other agents healthy

02Diagnose

Find the real cause

A release had gone out that day, and it looked guilty. Instead of rolling it back on a hunch, I compared the broken agent with the healthy ones. The real cause was a single badly formatted file.

agent-platform · claudeclaude code
> Only support-agent fails. What does it have that the healthy agents do not?
⏺ Bash(git diff --stat previous-release..this-release -- skills/)
(empty: the deploy did not touch skills)
⏺ Search logs("parse frontmatter", 3 days, by agent)
support-agent only
⏺ Its skill header has a second ": " in the description line. YAML cannot parse it, so the engine fails the whole agent.

03Reproduce

Prove the bug, then fix it

I wrote a test that reproduces the bug and watched it fail. Then the fix: a bad file is refused when it is saved, and skipped safely when it is read, so it can’t take an agent down again.

~/agent-platformzsh
% go test ./internal/skills -run TestBadHeader
--- FAIL: TestBadHeader_SkipsOnlyThatSkill
# refuse at save · skip at read
% go test ./internal/skills -run TestBadHeader
ok internal/skills 0.41s

04Ship

Ship it safely

It was tested and reviewed before merging. Then I checked it on dev before production, so the fix didn’t break anything else.

~/agent-platformzsh
% git push origin fix/skill-header
✓ all tests pass ✓ review approved
✓ merged ✓ works on dev
✓ released to production

05Watch

Make sure it stays fixed

After the release, I watched the same logs that raised the alarm until they stayed quiet, then told the team it was fixed.

service:engine env:prodlogs
⌕service:engine env:prod "parse frontmatter"
16:40:00INFOenginerelease v1.4.2 live
16:52:13WARNengineskill disabled: bad header agent=support-agent turn ok
no failed turns since release
service:engine status:errorlogs
⌕service:engine status:error
14:02:11ERRORengineturn failed agent=support-agent parse frontmatter: yaml: line 2
14:02:40ERRORengineturn failed agent=support-agent parse frontmatter: yaml: line 2
14:03:05ERRORengineturn failed agent=support-agent parse frontmatter: yaml: line 2
one agent · every turn · all other agents healthy
agent-platform · claudeclaude code
> Only support-agent fails. What does it have that the healthy agents do not?
⏺ Bash(git diff --stat previous-release..this-release -- skills/)
(empty: the deploy did not touch skills)
⏺ Search logs("parse frontmatter", 3 days, by agent)
support-agent only
⏺ Its skill header has a second ": " in the description line. YAML cannot parse it, so the engine fails the whole agent.
~/agent-platformzsh
% go test ./internal/skills -run TestBadHeader
--- FAIL: TestBadHeader_SkipsOnlyThatSkill
# refuse at save · skip at read
% go test ./internal/skills -run TestBadHeader
ok internal/skills 0.41s
~/agent-platformzsh
% git push origin fix/skill-header
✓ all tests pass ✓ review approved
✓ merged ✓ works on dev
✓ released to production
service:engine env:prodlogs
⌕service:engine env:prod "parse frontmatter"
16:40:00INFOenginerelease v1.4.2 live
16:52:13WARNengineskill disabled: bad header agent=support-agent turn ok
no failed turns since release

What the platform does

What agents can do on it.

  1. Share an agent without sharing your data. I built guest access across the services: per-guest permissions, separate credentials, isolated sessions.

  2. Agents remember how you like answers, across chats.

  3. People write in many languages. I built the resolver that picks each reply’s language, ran it in shadow mode, scored it on a labelled test set, then shipped it.

  4. Owners teach agents with small instruction files. The studio checks each one on save and points to the line to fix.

  5. Morning summaries without asking: a scheduled run posts a single clean message to the channel.

  6. Agents reach real apps (logs, code, docs, travel) through a single tool gateway, covered in case 02.

support-agent · chat
Which merchants failed checkout yesterday?
Mostly card declines, plus the odd timeout.

Share this chat

Only me
Specific peopleops teamsupport
Everyone in the workspace
Shared · each guest gets their own access and session
ops-agent · chat
Keep your answers short, in bullet points.
Saved to memory · answer style = short bullets
next day
What failed overnight?
• Some checkouts failed
• Most were card declines
• The timeouts need a look from memory
support-agent · chat
¿Cuál es el estado de mi pago?
old logic: Your payment is being processed.
resolver · user wrote Spanish → reply in Spanish
Tu pago se está procesando. ✓
query-logs.md
1  ---2  name: query-logs3  description: Query logs: by user3  description: "Query logs: by user"4  ---
ops-agent · schedule
RunEvery weekday
At09:00
Post to#ops
09:00 · running…
ops-agent
Morning summary: failed checkouts are down, and the timeouts need a look.
support-agent · toolkits
Observabilitylogs · metricsConnectConnectingConnected
Coderepositories · pull requestsConnectConnectingConnected
Knowledgecompany docsConnectConnectingConnected
Travelflights · staysConnectConnectingConnected

Every connection runs through the same gateway. That’s case 02 →

Working with the model

The AI part isn’t just calling a model.

How the model runsWhat the model readsWhat it writes
  1. mainfixes and evals, each tested and reviewed
  2. fix/clear-provider-errorsClear failure reasons

    Every provider error used to show as the same vague failure. Each now maps to a precise reason, so people can see what went wrong.

  3. eval/model-not-promptModel, not prompt

    Agents ignored instructions and every prompt rewrite failed differently. A small A/B with the prompt held fixed pointed at the model, and the team changed the default.

  4. fix/untrusted-fence-on-trimUntrusted text stays untrusted

    A path that shortened long tool results skipped the fence that marks them as data. It uses the fence now.

  5. eval/stale-labelsWhen the answer key is wrong

    A score dropped after a rule change. The test labels still used the old rule, so I separated real regressions from stale labels and model noise.

  6. fix/internal-context-off-channelsInternal instructions stay internal

    A filter stops the agent’s own instructions leaking into replies. Its first version matched words and wrecked good replies in testing, so I rebuilt it to match the leaked section’s shape.

  7. feat/reply-language-resolverThe right reply language

    The resolver ran in shadow mode beside the old logic and was scored on a labelled test set before it went live.

How I ship a change like this

engine · claudeclaude code
> One path that trims tool results drops the untrusted fence. Find it and fix it, failing test first.
⏺ Bash(git worktree add ../engine-fix -b fix/untrusted-fence-on-trim)
⏺ Agent(research): find every path that shortens tool output
⏺ Agent(skeptic): try to break the fix
FAIL trimmed result is missing its fence
ok fence kept after trimming
⏺ Agent(reviewer): review the diff
⏺ Ready for your review. Branch pushed, PR opened.
  • worktree an isolated copy of the repo per task, so parallel work never collides
  • subagents separate AI agents that research, challenge and review, so the writer never grades its own work
  • me I set the task and the bar, and I read every line before it ships

Things that broke

And why they stay fixed.

INC-01 · engine · scheduled runsResolved

A job said “ok” after doing part of its work

✕ a scheduled job stopped early and still logged success

✓ cause found: the step limit, not the rate limit

Postmortem

A scheduled job hit its step limit partway through and still recorded success. The chat app’s rate limit looked guilty; the real cause was the step limit. A job that says ok when it isn’t is worse than one that fails loudly.

INC-02 · tool gatewayResolved

A release broke credentials for ordinary agents

✕ tool calls lost their credentials

✓ rolled back the same day; the missing test now exists

Postmortem

A gateway change refused calls without an identity, on the premise that every caller sends one. Some callers never did. The release was rolled back the same day, and the test that was missing covers exactly that caller: a call that arrives without an identity.

INC-03 · engine · model catalogueResolved

My own change caused a retry loop

✕ an empty model name was rejected and re-sent every few minutes

✓ my mistake, found and fixed

Postmortem

A catalogue row was missing, so my change sent an empty model name. The engine rejected it, and the request was re-sent every few minutes. It was my mistake. I found it, fixed it, and now check for missing rows before I trust a lookup.

INC-04 · engine · chatResolved

A message vanished when a turn stopped early

✕ the user’s message was never saved

✓ save the message the moment the turn is accepted

Postmortem

When a turn stopped before the model ran, the user’s message was never written, so it disappeared from the chat. The message is now saved the moment the turn is accepted, and tests cover both early-failure paths.

Lessons

  1. 01

    Prove the cause before the fix

    The day’s release looked guilty, and rolling it back would have felt productive. Comparing the broken agent with the healthy ones found the real cause instead.

  2. 02

    Refuse bad input where it enters

    Check a file when it is saved, and survive it when it is read. A bad file should cost a single skill, not a whole agent.

  3. 03

    Ship risky logic in the shadows first

    The language resolver ran beside the old logic and was scored on a labelled test set before it replaced anything.

  4. 04

    Measure the model, not just the code

    A small A/B with the prompt held fixed showed the model was ignoring instructions. The team changed the default model on that evidence.