AI: Decision Models

2026-10-12

The question

A lot of LLM calls in my code are not really "chat". They are small decisions: which branch, which category, is this urgent, yes or no. For that I usually ask a big model to answer in JSON and then hope the JSON is valid. So the question is simple: is a model that only makes typed decisions a better tool for these small steps than a general LLM?

Chat API vs. Decision API

A decision model does not write free text. You give it a state and some typed questions. It returns typed answers with probabilities.

Chat APIDecision API
Inputmessagesstate + named questions
Outputtext (maybe JSON, if the model behaves)one of the predefined values + probabilities
Use in codeparse, validate, retryif answer.choice == "shipping": ...

The idea comes from Daniel Kahneman: System 1 is fast and intuitive thinking, System 2 is slow and deliberate. A decision model is the System 1 part: fast, cheap, no reasoning text.

What is different

Question types

There are three types. One request can mix them, and each question is answered against the same state.

TypeAsksReturns
choicepick one optionoption + probability per option
scorerate on an ordered scalevalue on the scale (also between levels) + probability per level
yes/nois this statement true?value between 0 and 1

A support ticket as example. The request (shortened):

{
  "state": "My order arrived damaged and nobody replied for a week.",
  "questions": {
    "department": {
      "type": "choice",
      "criteria": {
        "billing": "Payment or subscription issues",
        "shipping": "Delivery or product condition problems",
        "sales": "Pricing or account questions"
      }
    },
    "is_urgent": {
      "type": "yes/no",
      "instructions": "The message conveys urgency"
    }
  }
}

The answer:

{
  "department": {
    "choice": "shipping",
    "probabilities": { "billing": 0.05, "shipping": 0.82, "sales": 0.13 }
  },
  "is_urgent": 0.95
}

The exact field names differ from provider to provider. The shape is the same.

Confidence decides the next step

The calibrated probability is the interesting part. The code can decide by itself when the small model is good enough.

+-----------+  state + questions  +----------------+
| your code | ------------------> | decision model |
+-----------+                     +----------------+
                                          |
                                          v
                                  +----------------+
                                  |  confidence    |
                                  |  high enough?  |
                                  +----------------+
                                     |          |
                                 yes |          | no
                                     v          v
                          +------------+  +-----------------+
                          | use the    |  | ask a large LLM |
                          | answer     |  | or a human      |
                          +------------+  +-----------------+

        Figure 1: Escalation by confidence

Most cases stay fast and cheap. Only the unclear ones go to the slow, expensive path.

Good fit: routing, classification, scoring, yes/no checks. Not a fit: anything that needs generated text or code.

Without a decision API

A general LLM can do part of this today:

But these probabilities are not trained to be calibrated. And the model still generates text, so it is slower.

A test

I sent 12 support tickets to Jev via OpenRouter, each with four questions: team, urgency, is it a bug, churn risk. Thresholds in my own code decide the queue. All 12 took 600 ms in parallel and cost $0.0003 in total.

Ticket T08: team confidence 0.39 is below 0.60, so the code sends it to the review queue

Ticket T08 threatens to switch to a competitor and complains about a slow dashboard. Jev is unsure about the team (0.39), so my code sends it to a human. This is Figure 1 with real data.

What I learned:

Examples (October 2026)