Email Routing Agent
ShippedAn agent that automatically files inbound email attachments into the correct folder in the document management system.
The Problem
Our document inbox received hundreds of attachments every week. Every misfiled document became invisible to the AI knowledge workspace that depended on it. For a system that has to find the right information to chunk and analyze, invisible means useless.
The first version of this was a pure-AI approach. I gave the cheaper model the document summary and a list of 200+ possible folders, asked for a shortlist, then sent the shortlist to the more powerful model to make the final choice.
The result was a painful 37.5% accuracy rate.
The problem wasn’t the AI’s intelligence. It was its context. A general-purpose model has no way of knowing our internal logic — the institutional knowledge that dictates why some contracts go into the construction loan folder instead of the main contracts folder. It was trying to organize the library without knowing our Dewey Decimal System.
The pivot was to a deterministic Chain of Thought engine:
- A custom-trained classifier makes a simple, high-confidence call — what type of document is this?
- A focused LLM call extracts key metadata based on that type.
- Simple, rule-based code uses the structured data to build the exact folder path.
No guessing at the final step. Probabilistic inference happens where probabilistic inference belongs (classification and extraction), and deterministic logic handles routing. The lesson I carried into everything after: you cannot build a reliable knowledge system on an unreliable data foundation.
Essays about this build
The first version of our document routing agent hit 37.5% accuracy. The problem wasn't the AI — it was building probabilistic inference into a step that needed to be deterministic.