# Case 01: Building an AI agent platform

**What changed:** A single bad file can no longer take an agent down, and every provider error now names a precise reason.

Teams across the company built their own AI agents, with skills, memory and tools, inside their chat apps. I worked on what runs underneath: the engine behind every turn, the studio where agents are made, and the gateways to chat and tools.

Stack: Go, TypeScript/React, PostgreSQL, MongoDB, Docker, Kubernetes, AWS

## At a glance

- **My role:** AI engineer on the platform team. I took fixes and features end to end: root cause, failing test, review, release, then watching it.
- **Scope:** A multi-tenant agent platform used by teams across the company.
- **Stack:** Go · TypeScript/React · PostgreSQL · MongoDB · Docker · Kubernetes · AWS
- **Shipped:** Guest access across the services, a reply-language resolver, per-turn outcome records, and the fix for an outage caused by a single bad file.

## How I fix a bug in production

### 1. Report: Something breaks

A team reported that their agent had stopped replying. Before touching any code, I went to the logs.

```
service:engine status:error
14:02:11  ERROR  engine  turn failed  agent=support-agent  parse frontmatter: yaml: line 2
14:02:40  ERROR  engine  turn failed  agent=support-agent  parse frontmatter: yaml: line 2
14:03:05  ERROR  engine  turn failed  agent=support-agent  parse frontmatter: yaml: line 2
one agent · every turn · all other agents healthy
```

### 2. Diagnose: Find the real cause

A release had gone out that day, and it looked guilty. Instead of rolling it back on a hunch, I compared the broken agent with the healthy ones. The real cause was a single badly formatted file.

```
Only support-agent fails. What does it have that the healthy agents do not?
Bash(git diff --stat previous-release..this-release -- skills/)
(empty: the deploy did not touch skills)
Search logs("parse frontmatter", 3 days, by agent)
support-agent only
Its skill header has a second ": " in the description line. YAML cannot parse it, so the engine fails the whole agent.
```

### 3. Reproduce: Prove the bug, then fix it

I wrote a test that reproduces the bug and watched it fail. Then the fix: a bad file is refused when it is saved, and skipped safely when it is read, so it can’t take an agent down again.

```
go test ./internal/skills -run TestBadHeader
--- FAIL: TestBadHeader_SkipsOnlyThatSkill
# refuse at save · skip at read
go test ./internal/skills -run TestBadHeader
ok    internal/skills    0.41s
```

### 4. Ship: Ship it safely

It was tested and reviewed before merging. Then I checked it on dev before production, so the fix didn’t break anything else.

```
git push origin fix/skill-header
✓ all tests pass   ✓ review approved
✓ merged   ✓ works on dev
✓ released to production
```

### 5. Watch: Make sure it stays fixed

After the release, I watched the same logs that raised the alarm until they stayed quiet, then told the team it was fixed.

```
service:engine env:prod "parse frontmatter"
16:40:00  INFO  engine  release v1.4.2 live
16:52:13  WARN  engine  skill disabled: bad header  agent=support-agent  turn ok
no failed turns since release
```

## What the platform does

- **Chat sharing.** Share an agent without sharing your data. I built guest access across the services: per-guest permissions, separate credentials, isolated sessions.
- **Memory.** Agents remember how you like answers, across chats.
- **Reply language.** People write in many languages. I built the resolver that picks each reply’s language, ran it in shadow mode, scored it on a labelled test set, then shipped it.
- **Skills.** Owners teach agents with small instruction files. The studio checks each one on save and points to the line to fix.
- **Scheduled runs.** Morning summaries without asking: a scheduled run posts a single clean message to the channel.
- **Toolkits.** Agents reach real apps (logs, code, docs, travel) through a single tool gateway, covered in case 02.

## The AI part isn’t just calling a model

### How the model runs

- **Clear failure reasons** (`fix/clear-provider-errors`). Every provider error used to show as the same vague failure. Each now maps to a precise reason, so people can see what went wrong.
- **Model, not prompt** (`eval/model-not-prompt`). Agents ignored instructions and every prompt rewrite failed differently. A small A/B with the prompt held fixed pointed at the model, and the team changed the default.

### What the model reads

- **Untrusted text stays untrusted** (`fix/untrusted-fence-on-trim`). A path that shortened long tool results skipped the fence that marks them as data. It uses the fence now.
- **When the answer key is wrong** (`eval/stale-labels`). A score dropped after a rule change. The test labels still used the old rule, so I separated real regressions from stale labels and model noise.

### What it writes

- **Internal instructions stay internal** (`fix/internal-context-off-channels`). A filter stops the agent’s own instructions leaking into replies. Its first version matched words and wrecked good replies in testing, so I rebuilt it to match the leaked section’s shape.
- **The right reply language** (`feat/reply-language-resolver`). The resolver ran in shadow mode beside the old logic and was scored on a labelled test set before it went live.

## Things that broke

### A job said “ok” after doing part of its work

- Where: engine · scheduled runs
- Symptom: a scheduled job stopped early and still logged success
- Fix: cause found: the step limit, not the rate limit

A scheduled job hit its step limit partway through and still recorded success. The chat app’s rate limit looked guilty; the real cause was the step limit. A job that says ok when it isn’t is worse than one that fails loudly.

### A release broke credentials for ordinary agents

- Where: tool gateway
- Symptom: tool calls lost their credentials
- Fix: rolled back the same day; the missing test now exists

A gateway change refused calls without an identity, on the premise that every caller sends one. Some callers never did. The release was rolled back the same day, and the test that was missing covers exactly that caller: a call that arrives without an identity.

### My own change caused a retry loop

- Where: engine · model catalogue
- Symptom: an empty model name was rejected and re-sent every few minutes
- Fix: my mistake, found and fixed

A catalogue row was missing, so my change sent an empty model name. The engine rejected it, and the request was re-sent every few minutes. It was my mistake. I found it, fixed it, and now check for missing rows before I trust a lookup.

### A message vanished when a turn stopped early

- Where: engine · chat
- Symptom: the user’s message was never saved
- Fix: save the message the moment the turn is accepted

When a turn stopped before the model ran, the user’s message was never written, so it disappeared from the chat. The message is now saved the moment the turn is accepted, and tests cover both early-failure paths.

## Lessons

- **Prove the cause before the fix.** The day’s release looked guilty, and rolling it back would have felt productive. Comparing the broken agent with the healthy ones found the real cause instead.
- **Refuse bad input where it enters.** Check a file when it is saved, and survive it when it is read. A bad file should cost a single skill, not a whole agent.
- **Ship risky logic in the shadows first.** The language resolver ran beside the old logic and was scored on a labelled test set before it replaced anything.
- **Measure the model, not just the code.** A small A/B with the prompt held fixed showed the model was ignoring instructions. The team changed the default model on that evidence.

## More

- [All work](https://nityashree.in/index.md)
- Next: [Building MCP connections](https://nityashree.in/work/mcp-connections.md)