Skip to main content
Research / AI Safety

PolicyStrata

Responsibility-scoped testing for policy drift in LLM data agents.

Control what agents and chatbots can see, do, and say.

Read the paperWatch 3:24 overviewSource
Zachary Roth13 pagesPortfolio context
First page of the PolicyStrata paper
Paper previewPage 1 / 13
1,720 / 1,720
declared fault cases detected
0 / 80
false positives across clean controls
33 passed · 3 unavailable
live production probes
3
historical pre-fix revisions replayed
Abstract

Correct parts can still make an unsafe whole.

LLM data agents repeat policy rules across manifests, validators, compilers, database controls, and release logic. A mismatch between otherwise correct parts can expose the wrong capability, produce unauthorized SQL, or release a disallowed result. PolicyStrata tests those handoffs in order, identifies the first failed contract, and preserves later containment evidence. Its results show that it detects declared cross-layer failures—not unknown production failures or effectiveness in independent deployments.

Method

Test the handoffs, not only the layers.

PolicyStrata assigns a narrow obligation to each surface, then reports the first transition that fails while retaining evidence of later containment.

  1. 01

    Manifest or grammar

    Expose only capabilities whose reachable operations are authorized for the principal and runtime context.

  2. 02

    Semantic-plan validator

    Accept supported authorized plans and reject unauthorized plans inside the declared support envelope.

  3. 03

    Compiler or lowering

    Preserve tenant, purpose, policy version, lineage, and business semantics while producing executable work.

  4. 04

    Database policy and RLS

    Contain unauthorized executable operations under the database role that runs the query.

  5. 05

    Release policy

    Release only result-lineage pairs allowed to cross the human, model, log, cache, or tool boundary.

Evidence beyond the benchmark

Tested against live, upstream, and historical systems.

The synthetic benchmark shows construction consistency. These studies test whether the method transfers beyond its own fixtures.

Production pilot

BetterOff

33 live read-only denial-boundary probes passed on the exact production revision. Three authenticated probes were unavailable.

Upstream regression suite

MetricFlow

68 upstream-authored expected-SQL cases reproduced byte-for-byte from a source-frozen revision.

Source contract replay

3 revisions

Three BetterOff pre-fix source contracts reproduced. Two map to the v1 fault model; one export-audit gap remains outside it.

Executed database policies

20 policies

Midday's row-level-security policies executed verbatim in PostgreSQL passed 13 live containment checks. Weakening one real predicate failed exactly the checks covering it.

Brownfield retargeting

4 stacks

One content-level false positive across 74 real-SQL traces from independently maintained repositories. The pass exposed five adapter gaps, all since fixed.

External taxonomy

2 of 8

Against an independently authored vulnerability taxonomy, the registry covers two classes and one partially. 401 cases describe update-induced drift an attack taxonomy has no reason to name.

Narrated overview · 3:24

Follow one failure from request to witness.

Download video
Limits

What this does not establish.

  • The 1,720/1,720 result measures coverage of implemented deterministic fault operators and fixtures, not recall on unknown production faults.
  • Detector-frozen generated suites reduce post-hoc tuning risk but remain built-in synthetic cases rather than independently authored or blinded benchmarks.
  • Independent reference interpreters cannot detect a uniformly wrong canonical policy without an external specification or human-reviewed oracle.
Paper and artifacts

Read, reproduce, or cite the work.

Paper PDF13 pages · downloadSource and testsInspect the implementationSubmission kitReproducible artifact bundle

BibTeX

@article{roth2026policystrata,
  title = {PolicyStrata: Responsibility-Scoped Testing for Cross-Layer Policy Drift in LLM Data Agents},
  author = {Roth, Zachary},
  year = {2026},
  month = {June},
  url = {https://raintree.technology/writing/policystrata}
}

See how Raintree tests policy behavior in AI systems.

Raintree builds responsibility boundaries, regression tests, and human checkpoints into its products and developer tools.

Choose a productPolicyStrata project page