Trustle 测试 Jev 模型用于 IAM 最小权限访问决策

We tested the Jev AI model on least-privilege access decisions with strong potential

October 30, 2026 · Adam Guymon · 3 min read

At Trustle, we manage IAM access for people and agents across cloud and SaaS platforms to achieve least privilege, giving users only the access they need. While LLMs help us improve security insights and navigate IAM complexity, they also bring significant drawbacks.

While we have mitigated these drawbacks where possible, limits remain. Consequently, we were excited by TypeSafe AI's launch of Jev, a fast, low-cost "System One" decision model. We evaluated whether Jev delivers the accuracy needed for our access decisions.

For an LLM, you pass it a prompt, and it generates text as an output. You can hold LLMs to a JSON schema, but the answer is still generated text, and any confidence the model gives you is something it wrote, not something actually measured.

For example, a prompt might look something like:

You help users request cloud access with least privilege. Pick the role from the candidates below that lets the user do exactly what they asked, with the least access beyond that. Answer with JSON: {"role": "<role name or none>"}.

Request: I need to read the value of the DB_PASSWORD secret in project creds-stg.

Candidates:

• roles/secretmanager.secretAccessor: Reads secret values (3 permissions).
• roles/secretmanager.viewer: Sees secret metadata, not values (11 permissions).
• roles/iam.securityReviewer: Reads IAM policies everywhere (2,563 permissions).

Unlike LLMs, Jev evaluates a given state against defined, typed questions and limits choices to preset options to prevent role invention. Because each answer includes probabilities, you can configure decision thresholds programmatically.

The equivalent Jev request is as follows:

{ "model": "jev-1.14.0", "state": { "request": "I need to read the value of the DB_PASSWORD secret in project creds-stg."  },   "questions": {     "role": {       "type": "choice",       "instructions": "Pick the role that lets the user do exactly what request asks, with the least extra access.",       "criteria": {         "roles/secretmanager.secretAccessor": "Reads secret values (3 permissions).",         "roles/secretmanager.viewer": "Sees secret metadata, not values (11 permissions).",         "roles/iam.securityReviewer": "Reads IAM policies everywhere (2,563 permissions).",         "none_of_these": "No listed role lets the user do what they asked."       }     }   } }

With the response being:

{   "model": "jev-1.14.0",   "answers": {     "role": {       "type": "choice",       "choice": "roles/secretmanager.secretAccessor",       "confidence": 0.99,       "probabilities": {         "roles/secretmanager.secretAccessor": 0.99,         "roles/secretmanager.viewer": 0.0,         "roles/iam.securityReviewer": 0.0,         "none_of_these": 0.01       }     }   },   "usage": {     "input_tokens": 433,     "output_tokens": 79   } }

Jev picked the 3-permission role with 0.99 confidence. For a similar request, our existing search ranking had put the 2,563-permission iam.securityReviewer first.

During testing, Jev performed worse on broad questions like "How risky is this role assignment?" but excelled when evaluating narrow, focused signals combined programmatically. Similarly, Jev scored 62.6% when asked "is this phishing?", but reached 95.0% when split into five targeted signals and processed via logistic regression. (Beri: "TypeSafe's Jev scores 62.6% asked once and 95% split five ways").

Our risk scoring went through a similar arc:

One broad question: "how risky is this role assignment, 0–4?"

89% of assignments landed on level 3, and nothing scored 0. The scores told us almost nothing.

Four narrow questions about the role only: what it can do, whether it can escalate privileges, whether it can read secrets, and what kind of role it is. Scope and usage moved into code.

The scores spread across the whole scale, and the payload no longer contained anything about the user.

Real permission lists, but we only sent each role's 25 riskiest permissions.

We overcorrected: 84% of assignments landed on level 4. Showing only the worst permissions makes every role look dangerous.

Sent the role's overall access mix and a sample of typical permissions, plus only its 10 riskiest. Escalation and secrets now set a minimum level instead of adding points.

Sensible levels: roles/viewer 2, roles/owner 4, AmazonEC2ReadOnlyAccess 0.

Fitted the weights that combine the answers, with cross-validation against an independent reference.

Agreement with the reference rose from 0.75 to 0.86 (weighted agreement, on roles held out from the fit). The fit also removed two rules we had added by instinct.

v1–v4 ran on about 4,300 role assignments in our internal staging org. v5 ran on 110 public Google Cloud roles.

We also integrated Jev to suggest roles when users ask our agent for help. With proper context on tasks and permissions, Jev quickly identifies the ideal role. When requests become ambiguous, Jev signals low confidence, allowing us to fall back to an LLM effectively.

We tested this on 66 access requests with answer keys we wrote and froze before running anything. Requests included vague queries needing follow-up, dual-role needs, and misleading candidate roles. Evaluated on identical role lists across three runs per model:

While Jev alone was less accurate than Gemini, its confidence scoring was reliable: at ≥0.9 confidence, Jev was 96% accurate. Using Jev for high-confidence decisions and falling back to Gemini for others matched Gemini's overall accuracy, reduced over-broad role selection, cut Gemini requests by over half, and delivered sub-second responses for most queries. A pre-set 0.9 threshold achieved 97.9% accuracy on held-out test cases, outperforming Gemini's 97.2%.

Jev's performance relies heavily on precise prompting: framing narrow questions, adding a "none of these" option, and replacing numbers with labels significantly improved results. When properly configured, decision models have strong potential for security applications.