Notification Classifier
An LLM-powered triage system for exchange and vendor notifications: Claude classifies with an explicit confidence score, deterministic rules route, and doubtful cases go to a human review queue. 100% accuracy across 70 test notifications — zero lost.
Overview
Trading firms receive a constant stream of operational notifications — feed migrations, fee updates, regulatory changes, maintenance windows, licensing amendments, product launches. Each one must reach the right team immediately: a missed feed migration can mean lost market-data connectivity; a missed regulatory notice can mean a failed audit.
The classifier is a three-stage pipeline. Claude reads each free-text notification via forced tool use and returns a structured category, confidence score and reasoning. A deterministic router then applies one auditable rule: confidence at or above 0.8 auto-routes to the owning team; anything below — including API failures and invalid output — goes to a human review queue. A hand-rolled evaluation stage reports accuracy, per-category precision and recall, and the threshold trade-off.
The model is explicitly instructed to keep its confidence low when a notification plausibly fits two categories — trained transparency about ambiguity rather than confident guessing. That honesty is what makes the automation trustworthy: on the test set of 70 notifications (10 deliberately ambiguous), classification accuracy was 100%, eight of the ten ambiguous cases were caught by the review queue, and not a single notification was dropped.
Approach
Classify
Claude reads the raw notification and returns category, confidence and reasoning through forced tool use — structured output, with retry logic for transient API failures.
Route
Deterministic code applies the threshold: confidence ≥ 0.8 auto-routes to the mapped team; everything else lands in the review queue. Auditable, repeatable, no ML in the decision.
Review
Humans handle the queue — each entry arrives with the model's own reasoning attached, so a reviewer sees why the classifier hesitated.
Evaluate
Hand-rolled metrics — confusion matrix, per-category precision/recall, threshold sweep — written to a markdown report. The model is never used to grade itself.
The trade‑off, live
All 70 real classifications from the evaluation run, plotted by the model’s own confidence. Drag the threshold: everything to the right is auto-routed to its team, everything to the left lands in the human review queue. Accuracy stays at 100% at every setting — the threshold purely tunes review burden.
All six categories scored 100% precision and 100% recall. Of the ten deliberately ambiguous edge cases (ringed above), eight sit below the default 0.8 threshold — caught by the review queue exactly as designed.
Where the LLM
helps — and
where it doesn’t
Reading & categorising free text
Language understanding is the hard part — this is the only place the model is used.
Routing rules, thresholds, team mapping
Must be auditable and repeatable — plain deterministic logic, no ML, no magic.
Evaluation metrics
You don’t grade the model with the model — metrics are hand-rolled, no sklearn, no LLM.
Test dataset generation
Deterministic seeded templates keep the ground truth exact and avoid shared blind spots between generator and classifier.
Routing
map
Above the threshold, each category routes straight to its owning team. Below it — or on any API failure or invalid output — the notification lands in the review queue. The invariant is enforced in code: routed + queued = total. Nothing is silently dropped.