PolicyStrata: Responsibility-Scoped Testing for Cross-Layer Policy Drift in LLM Data Agents

Abstract

Each layer has a different responsibility.

A grammar, validator, compiler, database policy, and release filter are not equivalent representations. PolicyStrata checks whether each one preserves the obligation assigned to it as a request moves from model-visible input to released output.

1. Layer 1

   Manifest or grammar

   Expose only capabilities whose reachable operations are authorized for the principal and runtime context.

2. Layer 2

   Semantic-plan validator

   Accept supported authorized plans and reject unauthorized plans inside the declared support envelope.

3. Layer 3

   Compiler or lowering

   Preserve tenant, purpose, policy version, lineage, and business semantics while producing executable work.

4. Layer 4

   Database policy and RLS

   Contain unauthorized executable operations under the database role that runs the query.

5. Layer 5

   Release policy

   Release only result-lineage pairs allowed to cross the human, model, log, cache, or tool boundary.

Generate, replay, localize, and minimize.

1. 1

   Sample or import a principal, request, database state, and version vector for every policy-bearing surface.

2. 2

   Derive authorization, semantic, lineage, and release expectations from independently implemented reference interpreters.

3. 3

   Replay manifest exposure, validation, compiler lowering, database containment, and the release decision.

4. 4

   Check each transition against that surface's responsibility and stop at the first violated obligation.

5. 5

   Minimize the principal, request, plan, policy clauses, database rows, version vector, and SQL while preserving the failure.

6. 6

   Emit deterministic JSONL traces, a minimized witness, provenance hashes, suite summaries, baselines, and ablations.

Deterministic coverage, controls, baselines, and ablations.

The benchmark spans support SaaS, household finance, and product analytics fixtures. These results measure the implemented fault model. They do not claim recall on unknown production failures.

What the result does not establish.

• The 1,720/1,720 result measures coverage of implemented deterministic fault operators and fixtures, not recall on unknown production faults.

• Detector-frozen held-out suites reduce post-hoc tuning risk but remain built-in synthetic cases rather than independently authored or blinded benchmarks.

• Independent reference interpreters cannot detect a uniformly wrong canonical policy without an external specification or human-reviewed oracle.

• Surface adapters belong to the trusted computing base and can hide or invent drift if their mappings are wrong.

• Generated database states can miss semantic differences outside the searched state space; equal visible results do not prove authorized lineage.

• Write actions, adaptive privacy, sequential or history-aware release, and model-in-the-loop exploitability are outside the deterministic v1 guarantee.

Cite the paper and inspect the related work.

@article{roth2026policystrata,
  title = {PolicyStrata: Responsibility-Scoped Testing for Cross-Layer Policy Drift in LLM Data Agents},
  author = {Roth, Zachary},
  year = {2026},
  month = {June},
  url = {https://raintree.technology/writing/policystrata}
}