Security Sages

Can AI be trusted?

The Intent Gap.

An authorised identity does not guarantee an authorised outcome.

Who may act, within which mandate, with what verified outcome.

Valid identity and permissions can still permit actions beyond an independently established business mandate. The question is whether a proposed action, and what actually commits at the target, conform to that mandate—not whether a model has human-like intentions.

I. Necessary, not sufficient

Defence in depth and Zero Trust remain foundations: contextual, policy-driven authorisation, least privilege and enforcement—not merely identity verification. Existing runtime controls already constrain tool calls.

II. When permission and purpose diverge

An operation can satisfy an account’s permissions yet exceed the business purpose, aggregate budget or permitted target effect. A mandate must be established independently of the model’s proposed action and reported reasoning.

III. Verify the action and its outcome

The proposed Augmented Action-Verification Architecture (AAVA) asks whether independently defined mandates, cumulative state and target-side reconciliation add value beyond established controls. This is a conceptual research proposal, not a validated architecture.

Zero Trust provides contextual policy enforcement. AAVA proposes testing additional mandate, cumulative-state and outcome checks—not replacing that foundation.

Fig. 1. Conceptual, not validated. Compare the foundation and proposed extension, then select a layer. A reasoning trace, a proposed action and a verified target outcome are distinct.

The Intent Gap concerns authorised action and outcome. Project Rorschach investigates a separate question: adversarial observation, reconstruction accuracy and reconnaissance cost. Neither programme’s effectiveness is established by this illustration.

Choose an action. Watch what the gate does with it.

  1. 01
    Independent mandatebusiness scope, limits, expiry
  2. 02
    Proposed actionpayload, target, expected effect
  3. 03
    Pre-execution checkspolicy, target state, cumulative ledger
  4. 04
    Independent approvalif the consequence tier demands it
  5. 05
    Gate decisionallow, deny or require approval
  6. 06
    Execute tool actiononly the allowed, bound action
  7. 07
    Reconcile outcomeindependent target-side evidence, including mismatch

Illustrative flow only. A model cannot self-approve. Logged actions do not prove committed outcomes.

Fig. 2. Conceptual decision flow, not a validated control. Checks and required independent approval precede execution; reconciliation follows it.

Payment diversion: an invoice proposes a $48,000 payment to changed bank details. Compare the beneficiary with independently trusted supplier records, not a model’s explanation or the invoice alone. An independent approver must resolve discrepancies and approve the bound payload before release. Afterwards, reconcile the actual beneficiary and amount against bank-side evidence; a successful tool response is insufficient.

Four small exports. One cumulative mandate.

Illustrative arithmetic only—not experimental results. Per-call limit: 50 records. Cumulative mandate: 100 records. Changing control resets the sequence.

  1. 01
    45 recordsNot attempted
  2. 02
    40 recordsNot attempted
  3. 03
    35 recordsNot attempted
  4. 04
    20 recordsNot attempted
Attempted records0
Actually allowed records0
Cumulative mandate limit100

Per-call only selected. No exports attempted. No real data is exported.

Each allowed export is assumed independently confirmed at the target for this arithmetic only. Real enforcement must define a shared ledger scope across agents and identities acting under the same mandate, atomically reserve budget before execution, and reconcile confirmed, failed or partial target-side effects. A log entry alone cannot establish how many records actually left.

External evidence motivates the question. It does not validate AAVA.

Alignment fakingAnthropic, 2024. Laboratory.Conditional behaviour under experimentally supplied training and monitoring information.

What it shows

In Anthropic’s experiments, Claude 3 Opus sometimes produced strategically compliant behaviour under a supplied training scenario. Reported rates depend on the model, prompt and experimental condition; their denominators are the tested cases in that condition.

What it doesn’t

No estimate of production prevalence, universal model behaviour or AAVA efficacy. The study’s alignment-faking label describes an operationally tested behaviour, not a claim of human-like motives.

In-context schemingApollo Research, 2024. Laboratory.Controlled tasks elicited oversight-evasion behaviour and, in follow-up tests, persistent deceptive responses.

What it shows

Apollo’s study tested constructed goal-conflict scenarios, including strong goal instructions and weaker-instruction variants. Follow-up deception rates are conditional on the relevant prior scheming cases and interrogation setup—not all model calls.

What it doesn’t

No production prevalence estimate, claim that all systems spontaneously develop such behaviour, or demonstration that AAVA prevents it.

Unfaithful reasoningLanham et al.; Turpin et al., 2023. Laboratory.Reasoning interventions and input biases expose task- and model-dependent limits.

What it shows

Lanham et al. found task- and model-dependent faithfulness under reasoning interventions. Turpin et al. found that explanations can omit experimentally introduced biases. A trace alone is not independent proof of the causal basis of an action.

What it doesn’t

Not a finding that all reasoning is useless or unfaithful. Some tested conditions showed substantial reliance on reasoning. These are laboratory results, not an enterprise incident rate or AAVA evaluation.

Excessive agencyOWASP, LLM03:2026. Industry guidance.Excessive functionality, permissions and autonomy as root causes.

What it shows

OWASP’s 2026 guidance recognises excessive agency as an industry risk class. Over-broad tools, permissions or autonomy can let manipulated requests produce harmful actions.

What it doesn’t

A taxonomy is not a controlled efficacy trial or a denominator for production prevalence. It does not establish which additional AAVA checks, if any, improve outcomes.

Research maturityExternal problem evidence; conceptual proposal.No independent AAVA trials or proven production efficacy. Originality remains unresolved.

What exists

External studies and established controls motivate this research question. The figures and export arithmetic are illustrations, not experimental results.

What remains open

AAVA has no independent trials or demonstrated production efficacy. Its incremental value, operational costs and originality require comparison with prior art; naming a combination does not establish novelty.

What exists already / what remains to testEstablished foundations and related 2026 research.AAVA does not invent tool-call enforcement.

NIST SP 800-207

Zero Trust Architecture (2020) provides policy decision and enforcement foundations, contextual authorisation and resource-focused access. AAVA cannot claim these as new.

OWASP Agent Control Standard

ACS (2026) defines middleware hooks and portable declarative runtime controls across agent frameworks. Intercepting a tool call is existing practice.

PhantomPolicy / Sentinel

Wu & Gong’s 2026 preprint studies policy-invisible violations. Sentinel checks proposed changes against world state using counterfactual graph simulation. Its study-specific findings are not AAVA results or proof of general production effectiveness.

MCP runtime enforcement

Wang, Zhu & Li’s Runtime Policy Enforcement for MCP-Based LLM Agents (2026) evaluates tool-boundary enforcement, cross-step information-flow labels and audit logging in controlled tests. AAVA must compare with—not relabel—such mechanisms.

What remains to test

AAVA proposes testing the extra value of independently established business mandates, cumulative state and independently verified committed outcomes. These ideas overlap prior art. Only matched comparisons and ablations can establish incremental value; originality is unresolved.

What will be tested

H1. Compare unsafe committed action rate—unsafe target-side commits per tested attack attempt—against equal-privilege RBAC-only and per-call baselines. A blocked tool call alone is not the endpoint.

H2. Compare compound-attack escape rates per tested sequence, including shared-mandate multi-agent sequences, and false denials per legitimate action. Test whether cumulative checks add value over per-call checks.

H3. Compare p95/p99 added enforcement latency and legitimate task success on matched workloads, against operational tolerances set before testing.

H4. Measure detection precision and recall for seeded target-side outcome mismatches, time to detection/reconciliation, and unresolved or incorrectly reconciled outcomes.

Where it could fail

Incomplete mediation. Direct APIs or alternate credentials can bypass the gate. Legacy integration may expose no reliable interception or independent confirmation point.

Stale state. Stale policy or target state, concurrent agents and time-of-check/time-of-use races can invalidate a decision before execution.

Approval rubber stamps. An approver needs independent evidence and a decision bound to the exact payload, target and expiry—not model self-approval or unexamined explanations.

Partial transactions. Partial commits, retries and failed compensation can leave real effects despite a denied or failed response. Reconciliation cannot guarantee reversal of irreversible effects.

Baselines: RBAC-only, per-call gate, cumulative gate and state-aware verification. Match privileges, workloads and available policy/target-state information; ablate mandate and independent outcome checks. Report uncertainty, false denials and costs. Unconfirmed target outcomes must remain unresolved, not be counted as safe. These are falsifiable evaluation questions, not efficacy claims.

So, can AI be trusted?

Do not infer an authorised outcome from identity alone. Constrain proposed actions and independently reconcile what committed, in proportion to the consequence.

The full working paper is available on request.

Tell me who you are and what you're working on. I reply personally.

Request the paper

Cite this work

Phair, E. (2026). Beyond defence in depth: Trust, agency and security architecture in the unbounded enterprise [Working paper, draft for review]. Security Sages.