PayTrace

Problem & responsibility

Business problem: when payment completion drops, dashboards only say it dropped and logs only explain a single request — while error codes, promotion changes, and config releases may all land at once. Correlation is not causation. I independently designed and built everything from the event model and deterministic loss ledger to the diagnostic agent and its evaluation system.

System flow

1. Payment events
2. Unified funnel
3. Loss decomposition
4. Incident freeze
5. Read-only investigation
6. Evidence registry
7. Multi-root-cause diagnosis
8. Human disposition

Engineering focus

Deterministic Core

A nine-stage funnel and purchase-intent correlation are computed by deterministic code, so loss figures are exactly reproducible — the model never computes facts.

Governed Investigation

Tool calls pass a Before/After hook chain: failed parameter or incident-scope validation blocks execution outright, and large results are offloaded to artifacts so context keeps only summaries and references.

Evidence Contract

Every conclusion must bind evidence IDs, and conclusions are graded SUPPORTED / PARTIAL / UNKNOWN — UNKNOWN is a legitimate output that stops the model from inventing a complete-looking answer.

Evidence

Data

fully simulated data + fault injection · no real channels or personal data

Eval

two-track eval · response evaluation + trajectory evaluation

Stability

scored on pass^k continuous reliability, not pass@k

GT Leak

ground truth isolated · evaluator actively checks for leaks

Python · FastAPI · PostgreSQL · Redis · MCP · explicit FSM · fault injection · pytest

Design tradeoffs

1. Deterministic code computes facts; the model only organizes the investigation

   Why

   Loss figures must be exactly reproducible, while root-cause hypotheses have to converge gradually out of uncertain information — the two have different optimal solutions.

   Cost

   Every metric and every decomposition has to be modeled and tested explicitly instead of falling out of the model; any change to the data contract lands entirely in code.

2. Govern the diagnosis through a hook chain instead of letting the agent explore freely

   Why

   Unauthorized calls, context bloat, and process amnesia are inherent agent problems that a prompt reminder does not solve.

   Cost

   Degrade-on-read and block-on-write semantics have to be defined per tool, making tool onboarding clearly more expensive than exposing a query endpoint directly.

3. Treat UNKNOWN as a legitimate conclusion

   Why

   Producing a complete-looking answer on thin evidence is far more dangerous than admitting you do not know.

   Cost

   Reports will visibly say “no conclusion”, which requires human review to take over, and evaluation has to check specifically that UNKNOWN fires correctly rather than being skipped.

4. Fully simulated data with configurable fault injection — no real payment channels

   Why

   Real payment data carries privacy and compliance risk, and only fault injection with ground truth enables stable demos, automatic evaluation, and version regression.

   Cost

   It cannot prove business results in a real merchant environment, and the simulated distribution does not equal any actual payment platform — a boundary that has to be declared explicitly rather than glossed over.

What this system does not do

Deliberate non-goals. A boundary that is written down is a boundary that can be held.

• Does not execute payments and does not store card details or payment credentials
• Does not automatically change channel, routing, risk, or promotion configuration — high-risk actions stay with humans
• Does not connect to real payment channels or use real user data
• Does not present “not found within budget” as “no problem exists”
• Does not emit a root-cause conclusion that is not bound to evidence
• Does not read correlation as causation

Open questions

What I do not have an answer for yet. Happy to be asked about any of these.

• The fault-injection distribution is designed by hand, so it cannot cover failure shapes that have not yet been recorded in real production
• How to weight response evaluation against trajectory evaluation has no settled calibration method yet
• Feeding real bad cases back from production traces is a planned direction; today’s eval set is still mostly injected scenarios
• Simulated data can demonstrate diagnostic capability and relative improvement between versions, but not online business results — that limit is itself the project’s honest boundary