RuleArena

Problem & responsibility

Business problem: promotion, refund, points, and membership rules change faster than manual regression can follow. The agent finds paths humans did not think of; deterministic code proves whether a path is actually broken.

System flow

1. Rule change
2. RuleSpec contract
3. Human confirm
4. Adversarial search
5. Sandbox replay
6. Oracle verdict
7. Minimal counterexample
8. Release gate

Engineering focus

Anti Self-Proof

Candidate risk ≠ confirmed bug — the agent never touches ground truth; only a sandbox replay plus an oracle violation confirms an issue.

Bounded Exploration

No dynamic expression evaluation; a fixed set of domain primitives only, and unconfirmed rules never enter attack runs.

Regression Assets

Minimal counterexamples bind a full evidence chain, export to pytest in one click, and repaired versions replay historical cases.

Evidence

Eval Set

design · 24 cases, 16 dev + 8 hidden (isolated)

Mechanism

measured · 0 false positives · 0 ground-truth leaks · 3/3 same-version replays

Discovery

measured · 0–20%, below the ≥75% design gate

Gate

counterexample → pytest regression → release gate

FastAPI · PostgreSQL · Redis · Explicit FSM · Delta Debugging · pytest

Design tradeoffs

1. Adjudicate with a deterministic oracle, not an LLM judge

   Why

   Fund conservation and lifecycle invariants can be computed reliably; an LLM judge is fine for judging prose quality and wrong for deciding whether money is correct.

   Cost

   Every rule’s invariants must be modeled as executable assertions, and semantics outside that coverage can only be declared unknown — there is no fallback judgment.

2. Fast exploration in a reference simulator, real HTTP replay in an isolated sandbox

   Why

   Running only in a simulator is self-proof — the simulator saying something is broken does not mean the real system is broken. Routing everything through real HTTP is far too slow for search to unfold.

   Cost

   Two worlds have to be maintained with matching semantics, and the promotion path between them becomes an object that itself needs testing.

3. An explicit workflow/FSM drives the main flow; the agent only searches unknown paths

   Why

   The lifecycle here is finite and its states and failure semantics can be written down; handing the main flow to an orchestration framework hides the mechanisms that matter.

   Cost

   Gives up the setup speed a framework provides — retries, recovery, and state transitions are all implemented and tested by hand, at a clearly higher upfront cost.

4. Compile to RuleSpec with human confirmation instead of letting the agent read natural-language rules

   Why

   Ambiguity in a natural-language rule has to be confirmed by a person and frozen into a version, otherwise every downstream verdict rests on semantics that drift.

   Cost

   Rule onboarding gains a human step and throughput is bounded by confirmation capacity; RuleSpec’s expressiveness also bounds which rules can be verified at all.

5. Isolate ground truth so the agent can never reach it

   Why

   If the agent can read the answer key, a “bug” it finds may just be reciting the syllabus — a fake high score is more dangerous than a low one.

   Cost

   The hidden set can only be maintained on the evaluation side, so debugging cannot inspect failure details and problems have to be reconstructed from traces.

What this system does not do

Deliberate non-goals. A boundary that is written down is a boundary that can be held.

• Does not execute real payments, inventory, logistics, or merchant settlement
• Does not access production e-commerce systems
• Does not prove complex real-world concurrency or distributed transactions
• Does not model arbitrary domains freely — portability requires all four of finite actions, explicit state, an executable environment, and deterministic invariants
• Does not let the agent modify or ship business code
• Does not do formal verification and never claims a rule is absolutely safe
• Does not mass-generate rules and cases automatically
• Does not substitute an LLM judge for deterministic business adjudication

Open questions

What I do not have an answer for yet. Happy to be asked about any of these.

• Measured LLM strategy discovery rate is 0–20%, still below the deterministic BFS baseline (20–22%), and the single agent submitted no candidate within a 300s budget — this is the current focus
• The 90s → 300s change was a single-variable adjustment calibrated against measured p95 latency; a better budget curve is still open
• The 24-case platform benchmark proves the search and adjudication mechanism is trustworthy, and does not extrapolate to an arbitrary new rule — a single rule’s release verdict still needs rule-specific evidence
• A more realistic target system would not fix the search-layer gap, which is why external e-commerce integration was deferred until the search layer clears its bar