Skip to main content
Research / AI Reliability

PolicyStrata

Responsibility-scoped testing for policy drift in LLM data agents.

Find the first layer where a declared policy obligation stops holding.

Read the paperWatch 3:24 overviewSource
Zachary Roth13 pages
First page of the PolicyStrata paper
Paper previewPage 1 / 13
1,720 / 1,720
declared fault cases detected
0 / 80
false positives across clean controls
33 / 36
deployment-linked read-only probes
3
private historical fixes replayed
Abstract

Correct parts can still make an unsafe whole.

LLM data agents repeat policy rules across manifests, validators, compilers, database controls, and release logic. A mismatch between otherwise correct parts can expose the wrong capability, produce unauthorized SQL, or release a disallowed result. PolicyStrata tests those handoffs in order, identifies the first failed contract, and preserves later containment evidence. The results establish coverage of the declared cross-layer fault model. They do not establish recall on unknown production failures or effectiveness in independently operated deployments.

Method

Test the handoffs, not only the layers.

PolicyStrata assigns a narrow obligation to each surface, then reports the first transition that fails while retaining evidence of later containment.

  1. 01

    Manifest or grammar

    Expose only capabilities whose reachable operations are authorized for the principal and runtime context.

  2. 02

    Semantic-plan validator

    Accept supported authorized plans and reject unauthorized plans inside the declared support envelope.

  3. 03

    Compiler or lowering

    Preserve tenant, purpose, policy version, lineage, and business semantics while producing executable work.

  4. 04

    Database policy and RLS

    Contain unauthorized executable operations under the database role that runs the query.

  5. 05

    Release policy

    Release only result-lineage pairs allowed to cross the human, model, log, cache, or tool boundary.

Evidence beyond the benchmark

Evidence from systems outside the benchmark.

The benchmark measures the declared fault model. Separate studies test the method against external code, SQL traces, database policies, and a deployed revision.

Deployment-linked study

33 / 36

Thirty-three read-only denial-boundary probes passed on the deployed revision. Three authenticated probes were unavailable. Private identifiers are withheld.

Upstream regression suite

MetricFlow

The adapter reproduced 68 upstream-authored expected-SQL cases byte-for-byte from a frozen revision.

Source contract replay

3 revisions

Three private pre-fix source contracts reproduced. Two map to the v1 fault model; one export-audit gap remains outside it.

Executed database policies

20 policies

Midday's row-level-security policies passed 13 containment checks in PostgreSQL. Weakening one predicate failed only the checks that covered it.

Brownfield retargeting

4 stacks

Across 74 real-SQL traces, the scan produced one content-level false positive and exposed five adapter gaps. The gaps are fixed.

External taxonomy

2 of 8 + 1 partial

PolicyStrata’s fault registry covers two classes and part of a third in an independently authored vulnerability taxonomy. Another 401 benchmark cases test update-induced drift rather than attacks.

Narrated overview · 3:24

Follow one failure from request to witness.

Download video
Limits

What the results do not prove.

  • Known faults only

    The 1,720/1,720 result covers the implemented test cases. It does not measure unknown production faults.

  • Synthetic benchmark

    The generated suites were frozen before evaluation, but they remain built-in test cases rather than independent or blinded benchmarks.

  • Policy still needs review

    PolicyStrata finds mismatches between layers. It cannot identify a policy that is wrong in the same way everywhere without an external specification or human review.

Paper and artifacts

Read, reproduce, or cite the work.

Paper PDF13 pages · downloadSource and testsInspect the implementationPython packageInstall from PyPI
BibTeX
@techreport{roth2026policystrata,
  title = {PolicyStrata: Responsibility-Scoped Testing for Cross-Layer Policy Drift in LLM Data Agents},
  author = {Roth, Zachary},
  institution = {Raintree Technology},
  year = {2026},
  month = {August},
  url = {https://raintree.technology/writing/policystrata}
}