The Gate
How one claim is verified — decompose, retrieve, NLI, a calibrated diverse ensemble, compliance rules — into a green/amber/red claim ledger. · 4 min read
The Gate answers one question for every claim in a message: is this entailed by an approved source, and is it permissible? It produces a claim ledger where each claim is green (entailed and permissible), amber (uncertain or needs a disclaimer), or red (unsupported or impermissible). A red claim is blocked and never becomes a sendable variant. The code lives in pipeline/gate/gate.py.
The stages of one verification
- Decompose — a rendered message is split into sentences, and each one is kept as a checkable claim only if it covers an approved library claim by token overlap, or asserts a superlative/guarantee. Greetings and CTAs are skipped (pipeline/gate/decompose.py).
- Retrieve — a BM25 lexical search over the Library's current source sentences returns the top evidence sentence for the claim. The index is rebuilt whenever the Library version changes, so a changed source is seen (pipeline/gate/retrieve.py).
- NLI — a deterministic, number-aware entailment scorer compares claim to evidence; it is strict about unsupported numbers and absolutes (pipeline/gate/nli.py).
- Ensemble — a diverse judge ensemble votes: a coverage lens, a numeric lens, and a superlative lens. Numeric and superlative are veto lenses (pipeline/gate/ensemble.py).
- Calibrate — an isotonic (PAV) calibrator turns the raw entailment score into a confidence; unfit, it is the identity (pipeline/gate/calibrate.py).
- Rules — the compliance engine applies legal holds, text blocks, and disclaimers; it can only make a verdict more restrictive (pipeline/gate/rules.py).
- Ledger — the most-severe of the entailment verdict and the rule verdict becomes the claim's final color.
The judge is one ensemble member, not the verifier
The LLM judge is not the decision-maker. The verdict is computed by combining the deterministic NLI score with the ensemble votes: support lenses (coverage, NLI, and any LLM judges) are averaged, then capped by the minimum of the numeric and superlative veto lenses (see Gate._combine). So a claim that drifts a number or invents a superlative is held down regardless of how confident any single judge is. LLM judges (Ollama / Claude) only run in the optional rich profile, only on uncertain claims, and only ever add votes — they cannot overrule a veto.
Idempotent and claim-level cached
Every verdict is cached on the key (claim_id, source_version, rules_version) (pipeline/common/cache.py). Each unique claim is Gated once, not once per recipient — the website channel reads the same cache the email channel wrote. A second verification of the same claim is served from cache with no recomputation, which is exactly what tests/test_gate.py::test_claim_is_gated_once asserts.