Blog • Agent Security

Why an AI Agent Cannot Trust JSON Schema Alone: Validation, Untrusted Input, and the Permission Decision

A tool call that passes JSON Schema only shows that the arguments have the right shape. Sending mail, changing files, or leaving the machine needs a second JSON document: a permission decision. If decision is not allow, the call stops.

Why output needs a shape is in why AI needs JSON Schema. Who is responsible when many tool servers are connected is in MCP security and permissions. This article is only the decision after the shape passes: the allowlist, the grant, and allow / deny / ask.

• Schema only answers whether the shape is right
• The tool name must already be on your allowlist
• The permission decision is a separate JSON document
• If the grant lacks the capability, do not run the tool

What the schema proves

JSON Schema checks types, enums, and required keys. See JSON Schema. It does not prove this session may send mail, and it does not prove you reviewed this tool name.

{
  "tool": "send_email",
  "arguments": {
    "to": "ops@example.com",
    "subject": "status",
    "body": "ok"
  }
}

When to, subject, and body are strings, an argument schema only sees types. It does not see that this session's grant has read_inbox and not send.

The decision is another JSON document

The MCP tools specification requires input validation and a way for a person to deny the call. See MCP Tools. After the argument JSON passes, write a decision. toolAllowed means this tool name is on your allowlist. grant is the capability this session already has. decision may only be allow, deny, or ask.

{
  "tool": "send_email",
  "argumentsValid": true,
  "toolAllowed": true,
  "grant": ["read_inbox"],
  "decision": "deny",
  "reason": "grant_missing"
}

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "required": ["tool", "argumentsValid", "toolAllowed", "decision", "reason"],
  "properties": {
    "tool": { "enum": ["send_email"] },
    "argumentsValid": { "type": "boolean" },
    "toolAllowed": { "type": "boolean" },
    "grant": {
      "type": "array",
      "items": { "enum": ["read_inbox", "send"] }
    },
    "decision": { "enum": ["allow", "deny", "ask"] },
    "reason": {
      "enum": ["ok", "grant_missing", "tool_not_allowlisted", "arguments_invalid"]
    }
  }
}

This schema stops a decision outside the three values, a non-boolean argumentsValid, a missing decision, and a grant value outside the list. It cannot say that allow requires both toolAllowed and the needed grant. Those rules stay in the app.

A pass you still reject

Change the decision above so decision is allow and reason is ok, and leave grant as read_inbox. This schema passes. Passing is not permission to send mail. The app rule is: decision may be allow only when argumentsValid and toolAllowed are both true and grant contains send. If any one is missing, set deny or ask and do not call the tool.

Where three cases stop

Annotations on a tool description are hints. The specification says annotations from an untrusted source are not a safety control you already enforced. See tool annotations are not a contract. What actually stops the call is your schema and this decision.

Untrusted text can lead a model to propose a call, and an untrusted tool description can make a dangerous capability look like an ordinary tool. Either can produce JSON with a valid shape. The defense is not to reproduce that text. Put the tool on the allowlist first, pass the call through the schema, then use the grant to choose allow, deny, or ask. A person can still deny the call.

How this differs from the other articles

Stop a bad allow in JSONNote

1. 1
   Pick the right operation

   Open JSON Schema. Set Operation to Check with Schema. Paste the permission decision in the first field and the schema above in the second. decision may only be allow, deny, or ask. Each grant entry may only be read_inbox or send.

2. 2
   Reject four bad documents

   Set decision to yes, set argumentsValid to the string true, delete decision, or set grant to send_all. Each of those fails. Setting decision to allow, reason to ok, and leaving grant as read_inbox passes. After it passes, change it back to deny or ask and do not call the tool.

3. 3
   Diff deny against allow

   Open JSON Compare Diff. Paste the deny decision in the first field. In the second, change only decision to allow and reason to ok. grant stays read_inbox. Both pass the schema. The app rule stops the send, not this shape check.

Common questions

Can the tool run once the schema passes?

No. The schema only proves the argument shape. You still check whether the tool is allowlisted and whether the grant includes the capability. If the decision is not allow, do not call the tool.

If the permission decision passes its schema, can you set allow?

Not from that alone. decision allow, reason ok, and a grant of only read_inbox is still a valid shape. The app must also see toolAllowed true and the needed capability in grant. If either is missing, it is not allow.

What do prompt injection and tool poisoning mean here?

They are names of two risks, not steps this article reproduces. One is untrusted text leading a model to propose a call. The other is an untrusted tool description that makes a dangerous capability look ordinary. Both can yield JSON with a valid shape. What stops them is the allowlist, the schema, and the grant, not rewriting that text.

Is this the same article as why JSON Schema is needed?

No. Those articles explain why output needs a shape. This one starts after the shape has passed: the allowlist and the grant can still make decision deny or ask.

What is the difference between ask and deny?

deny means do not call this time. ask means show the arguments to a person first; if the person refuses, still do not call. Neither is allow. A person in the loop is the deny capability stated in the MCP tools specification.

Summary

JSON Schema stops tool arguments with the wrong shape. The permission decision stops a call whose shape is already right but whose capability is wrong. Keep both JSON documents, and the execution has a gate.

The decision: a schema pass only means the shape is right; put the tool on the allowlist first; the grant must include this capability; if decision is not allow, do not call; if decision is allow but the grant is short, change it to deny or ask even though the shape passed.

Back to the blog