
A JSON schema Is not an AI governance layer
The AI governance argument is happening one layer above the code that actually stops your agent.
以下正文同步自 Stack Overflow Blog,版权归原站所有,已转换为易读排版。
October 28, 2026A JSON schema Is not an AI governance layer
The AI governance argument is happening one layer above the code that actually stops your agent.
In the space of two weeks this September, the argument about who should govern AI produced four different answers, all of them from the people who build the systems.
Demis Hassabis floated an industry-funded oversight body, modeled loosely on FINRA, to test frontier models before deployment. Dario Amodei called on labs to slow frontier development and give independent evaluators genuine access, and days later Anthropic and Accenture committed at least $2 billion over five years to embedded evaluation. OpenAI started publishing reports on unexpected model behavior. And according to the Wall Street Journal, Mark Zuckerberg, Elon Musk and Jensen Huang each spoke with President Trump and persuaded him to drop the oversight body, arguing that a common regulator would protect the lead of the biggest labs, and that companies can secure their own systems without one.
Whichever of these positions you agree with, none of them talks about the code that decides whether the agent in your application is allowed to send an email, rotate a key, merge a branch or move money.
That decision is not made by a lab, and it is not written in a policy; it is made by a function in your own code, and that function returns true or false.
I want to show you a bug in that function, and it’s not a thought experiment: an independent stress test found it in PIC, an open protocol I maintain for checking agent actions before execution. The schema accepted an uppercase SHA-256 digest, the reference runtime quietly normalized it and carried on, and the specification had never said whether that representation was valid. No cryptography failed. The protocol simply had more than one defensible meaning, which is harder to spot than broken cryptography, and I suspect many agent deployments running today have the same problem.
Stack Overflow’s April 2026 pulse survey of 1,100 developers and technologists found that 60% of them block agents from making unapproved system changes. That makes sense, but it leaves out the difficult part: what counts as approved, and will every component in the path agree on the answer?
Here is an agent proposing a payment. It supplies the tool it wants to call, the inputs that shaped the decision, and the evidence behind its claim that the payment is authorized.
{ "action": {"tool": "payments.transfer", "args": {"amount": 500}}, "impact": "money", "provenance": [ {"id": "invoice", "trust": "trusted"}, {"id": "invoice", "trust": "untrusted"} ], "claims": [ {"text": "The invoice was approved", "evidence": ["invoice"]} ] }
A JSON Schema will confirm that provenance is an array, that every entry carries an identifier and a permitted trust value, and that the required fields are present. Unless the protocol adds a rule of its own, this payload is valid. The identifiers are not unique, and the evidence reference now points at two different entries.
Suppose the policy is simple: a payment may run only if the evidence behind its claim comes from a trusted source. (For now, take the trust labels in the payload at face value; I come back to who gets to set them later.) To apply that policy, the verifier has to work out which entry “invoice” refers to, and two reasonable engineers resolve it differently.
Implementation A, Python: stop at the first match
def resolve(provenance, evidence_id): for entry in provenance: if entry["id"] == evidence_id: return entry return None
// Implementation B, TypeScript: index by id, last write wins const index = Object.fromEntries(provenance.map(e => [e.id, e])); const resolve = (id) => index[id];
Neither lookup is a bug on its own; both are normal, idiomatic code, and the specification never said which entry should win. Given the payload above, A resolves the evidence to the trusted entry and the payment executes, while B resolves it to the untrusted entry and the payment is blocked. Both verifiers run the same protocol version, with the same policy and the same trusted keys; the only difference is how each one looks up an id. So the same bytes, checked against the same schema, give two different results, and an attacker only needs to know which service is on the other end of the call.
The schema did its job, but that job is limited. It checks the shape of the payload: that fields are present, have the right type, and hold allowed values. It can even express some rules between fields, such as “if this field is present, that one is required”. What it can’t express is a rule like the one this payload needs: the “invoice” cited as evidence in claims has to point to exactly one entry in provenance. Standard JSON Schema has no way to say that a value in one place must match exactly one value somewhere else in the document; and uniqueItems doesn’t help either, because the two entries have different trust values, so the schema sees them as two different items. Whether the payment goes through depends on that link, and the schema leaves it to each verifier to work out.
Security people have seen this many times: two components read the same input in two different ways, and the difference between them is where the vulnerability is. HTTP parameter pollution is one example, where a proxy reads the first id and the application reads the last. XML signature wrapping is another: the verifier checks the signature on one element, while the parser reads a different one. Then there is JWT alg confusion, where the token itself says which algorithm should be used to check it. And with HTTP request smuggling, a proxy and the server behind it disagree about where one request ends and the next one begins.
This kind of bug has been around for about twenty years. What we have learned from it is fairly simple: when two implementations need to agree on what the same bytes mean, any detail the specification leaves open can be read in two ways, and sooner or later an attacker will take advantage of that.
Agent protocols are now running into the same problem, and PIC did too: the stress test I mentioned found these gaps, and we closed them by tightening the specification and adding each case to the conformance tests I describe below. Most of these protocols are only a year or two old, and they haven’t yet been tested the way token and signature formats have been over the years; so I think a governance layer written this year should be checked with the same care we would give a new token format.
Developers normalize input constantly. We trim whitespace, lowercase identifiers, repair Base64 padding, accept alternate encodings, all because rejecting data that is obviously recoverable feels hostile. That instinct is right in a search box and dangerous at an authorization boundary.
Suppose a policy permits exactly one tool:
expected_tool = "payments.transfer" proposed_tool = "payments.transfer " # note the trailing space
Compare those strings exactly and the proposal fails. Call strip() on both first and it passes. Neither behavior is irrational on its own; the defect is that individual implementations get to choose.
With signatures, the problem is more serious than it looks. A SHA-256 digest is usually rendered as hexadecimal, and uppercase A through F denote the same number as lowercase a through f, so lowercasing the input looks harmless. But it’s not, because signatures cover bytes rather than numbers. If the producer signed a payload containing the uppercase form and the verifier lowercases before constructing its canonical payload, verification fails on an authentic proposal.
Flip it around and it gets worse: a verifier that checks the signature over the original string and then normalizes the value before applying the policy is no longer enforcing exactly what was signed. For a hex digest, lowercasing gives the same number, so nothing changes; but the same habit applied to other fields can change which policy rule matches (a trimmed or lowercased tool name, for example, can match an allow-list entry that the signed name would not), and at that point the policy is acting on a value the signer never signed.
Base64 has the same problem: some libraries tolerate embedded whitespace, some restore missing padding, some silently translate the URL-safe alphabet, while others reject all three. When a signature decides whether an action may execute, those choices made by your Base64 library become part of your authorization policy, even though nobody on your team wrote them or reviewed them.
It helps to separate two decisions here. The first is how the signed payload is turned into bytes. Standards such as RFC 8785 for JSON, XML c14n before it, and the deterministic encoding rules in COSE fix things like field order, number format, and escaping, so that every implementation signs exactly the same bytes; but they leave the contents of a string alone, so RFC 8785 keeps an uppercase digest uppercase. The second decision is which values the protocol accepts at all. Here you can either normalize them (for example, lowercase every digest before signing and before checking) or accept a single form and reject everything else; so uppercase hex fails, unpadded Base64 fails, and a tool name with a trailing space fails.
PIC does both. In its canonical attestation mode it signs a canonical serialization (it still keeps a legacy mode that verifies raw payload bytes), and for values it took the second route, for a simple reason: normalization is one more piece of code that has to be written again in every language that uses the protocol, and the history of c14n bugs suggests that different implementations of the same rules end up disagreeing, just as parsers do. With strict rejection, a payload in the wrong format is refused as soon as it arrives, with a clear error, instead of getting through and causing two verifiers to disagree later, when the problem is much harder to track down. If you are designing something similar, pick the approach you can describe completely in your specification; but do pick one, because otherwise the meaning of a signed proposal depends on whichever library happens to decode it.
The ambiguity that mattered most was not about encoding at all, but about who gets to declare an input trustworthy.
{ "id": "invoice", "source": "email_attachment", "trust": "trusted" }
That field is easy to serialize, but on its own it tells the verifier almost nothing it can rely on. The proposal is produced upstream of the verifier, frequently by the very agent requesting permission to act. Honoring the field at face value lets the requester supply its own authority, which is the classic confused deputy problem, only written in JSON.
Trust has to be derived inside the enforcement boundary. The producer may identify a source and attach evidence, while the verifier decides whether that evidence leads back to something the operator actually trusts, such as a known signing key.
effective_trust = "untrusted"
if signature_is_valid(evidence, operator_keyring): effective_trust = "trusted"
if proposal.impact == "money" and effective_trust != "trusted": return BLOCK
The difference between integrity (the data wasn’t changed) and authority (someone allowed to approve it actually did) is what matters most here. A matching hash proves the verifier received the bytes it expected, but it says nothing about who produced them, who approved them, or whether they should authorize a payment. An attacker can supply a file together with the correct hash of that same file; the integrity check passes, because nothing was changed, but nobody with any authority has approved anything.
A signature carries authority only when the verifier looks up the signing key in a list of keys the operator trusts (a list the proposal has no way to add to), and only when the signature covers the fields that matter: the tool, the arguments, the
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。