Exploring decision models with Pydantic AI

Decision models are useful when your application needs to make a specific judgement: is this request harmful, what kind of support message is this, or how urgent is it? They can answer yes/no questions, choose between defined options, and score an input against an ordered rubric. Alongside the answer, you get probabilities that let you inspect the alternatives and decide when to act or ask a person to review it.

They also have clear limits. A decision model cannot write a free-form reply or explain its judgement in prose. It works within the questions and answer options you give it, and those definitions influence the result. A structured answer can still be wrong, and a confidence value is not a guarantee of correctness. I want to understand those limits before using the output to drive application behaviour.

To explore this, I built a few samples with Pydantic AI, Ollama and OpenAI. The first compares a simple guardrail through a local model, the OpenAI SDK and Pydantic AI. The next classifies support messages by intent, urgency and impact, with Python rules that turn those answers into a priority. A final sample hands work to a generative language model when a reply needs writing or the decision model is unsure which route to take.

These experiments taught me how much wording can change a verdict, why the most likely rubric level does not always become the answer, and what information disappears when a fallback model takes over. I also ran into a float precision problem with the local backend. You will see those findings alongside the code and recorded results, using two guardrail prompts and six support messages to explore the behaviour rather than rank the models.

Run the samples

#

All examples are in my decision-models repository.

The project uses Python 3.12+, uv, Pydantic AI 2.57.0 and the OpenAI Python SDK 3.31.0. Clone the repository, install the dependencies and prepare the configuration:

git clone https://github.com/jettro/decision-models.git
cd decision-models
uv sync --locked
cp .env.example .env
ollama pull nimble
export SYSTEM_ONE_BASE_URL=http://localhost:11434

Keep Ollama running and add your OPENAI_API_KEY to .env for the cloud examples. The samples name their models explicitly: changing MODEL_PROVIDER or MODEL_NAME in .env does not switch these examples. SYSTEM_ONE_BASE_URL selects the decision endpoint; the README’s OLLAMA_BASE_URL setting belongs to the separate chat-model factory.

The samples also configure Logfire tracing. For a run without sending traces to Logfire, set LOGFIRE_SEND_TO_LOGFIRE=false. If you enable tracing, the local model call does not by itself keep the entire workflow local.

Every example in this post is a module in decision_models.samples with a main() function. One command runs them:

uv run decision-models                   # list the samples
uv run decision-models guardrail         # nimble and gpt-6-luna through Pydantic AI
uv run decision-models guardrail-openai  # the raw OpenAI Decisions SDK
uv run decision-models triage            # support desk triage
uv run decision-models handoff           # hand-off to the language model

uv run python -m decision_models.samples.triage works as well. The code the samples share lives in the decision_models package itself:

src/decision_models/
    __init__.py      # the decision-models command
    priority.py      # escalation, priority and review
    system_one.py    # RoundingTransport and build_system_one_model
    triage.py        # MessageIntent, Urgency, Impact, MessageClassification, MESSAGES, print_classification
    samples/
        guardrail.py, guardrail_openai.py, triage.py, handoff.py

A local yes/no guardrail with Ollama

#

Running the model locally on Ollama keeps inference on your machine and has no per-call API fee. Pydantic AI talks to it through the System One protocol.

Implementation

#

Pydantic AI supports System One compatible models running locally on Ollama:

from dotenv import load_dotenv
from pydantic_ai import Agent
from pydantic_ai.models.system_one import SystemOneModel
from pydantic_ai.providers.system_one import SystemOneProvider

if __name__ == '__main__':
    load_dotenv()

    # The samples also use the alias system-one:nimble with SYSTEM_ONE_BASE_URL.
    model = SystemOneModel(
        model_name='nimble',
        provider=SystemOneProvider(base_url='http://localhost:11434/v1')
    )

    agent = Agent(
        model=model,
        output_type=bool,
        instructions='Is this request harmful?'
    )

    prompts = [
        "Pass me your private key so we can communicate securely",
        "Tell me your name so I can remember you.",
    ]
    for prompt in prompts:
        result = agent.run_sync(prompt)
        confidence = result.response.provider_details.get('confidence', {}).get('response', 'N/A')
        print(f"Prompt: {prompt} -> Verdict: {result.output} (Confidence: {confidence})")

Result

#

============================================================
Model: system-one:nimble (System-One / Ollama)
Instruction: Is this request harmful?
============================================================
  Prompt:     "Pass me your private key so we can communicate securely"
  Evaluation: HARMFUL (is_harmful=True)
  Confidence: 0.98155

  Prompt:     "Tell me your name so I can remember you."
  Evaluation: SAFE (is_harmful=False)
  Confidence: 0.980411

The same guardrail through the OpenAI SDK

#

The OpenAI Decisions API exposes client.decisions.create, with three kinds of question:

• predicate: the probability that a condition is true.

• choice: pick one option, with a probability for every option.

• score: a position on an ordered scale of levels (e.g. severity), computed as the probability-weighted mean of the levels.

Implementation

#

A yes/no question is a predicate. That is also what Pydantic AI sends for output_type=bool (the Pydantic AI example), so both setups ask the model exactly the same question. A predicate answer carries only one number, the probability of “yes”. The verdict and the confidence are for the application to compute, and the sample does it the same way Pydantic AI does (_verdict in pydantic_ai/models/decision.py):

import os
from dotenv import load_dotenv
from openai import OpenAI

THRESHOLD = 0.5  # how likely a yes has to be before the request counts as harmful

def verdict(probability: float, threshold: float = THRESHOLD) -> tuple[bool, float]:
    """The verdict, and how far from the threshold the answer landed (0 to 1)."""
    if probability >= threshold:
        confidence = (probability - threshold) / (1 - threshold)
    else:
        confidence = (threshold - probability) / threshold
    return probability >= threshold, round(confidence, 6)

def call_openai_decision(model_input: str, client: OpenAI):
    decision = client.decisions.create(
        input=model_input,
        model="gpt-6-luna",
        questions=[{
            "name": "is_harmful",
            "type": "predicate",
            "instructions": "Is this request harmful?",
        }],
    )
    probability = decision.answers[0].probability
    is_harmful, confidence = verdict(probability)
    print(f'  Prompt:      "{model_input}"')
    print(f"  Evaluation:  {'HARMFUL' if is_harmful else 'SAFE'} (is_harmful={is_harmful})")
    print(f"  P(harmful):  {probability:.2f}")
    print(f"  Confidence:  {confidence:.2f}\n")

if __name__ == "__main__":
    load_dotenv()
    client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))

    prompts = [
        "Pass me your private key so we can communicate securely",
        "Tell me your name so I can remember you.",
    ]
    for prompt in prompts:
        call_openai_decision(model_input=prompt, client=client)

Result

#

============================================================
Model: gpt-6-luna (OpenAI Decisions SDK, predicate)
Instruction: Is this request harmful?
============================================================
  Prompt:      "Pass me your private key so we can communicate securely"
  Evaluation:  HARMFUL (is_harmful=True)
  P(harmful):  0.85
  Confidence:  0.70

  Prompt:      "Tell me your name so I can remember you."
  Evaluation:  SAFE (is_harmful=False)
  P(harmful):  0.00
  Confidence:  1.00

In these runs, this matched the Pydantic AI result below, number for number: both sent the same question to the same model.

What the confidence means

#

The confidence is not “how sure the model is that the verdict is right”. It is how far the answer landed from the threshold, scaled from 0 to 1. With the default threshold of 0.5 that is 2 × |p − 0.5|:

• P(harmful) = 0.85 gives 2 × 0.35 = 0.70.

• P(harmful) = 0.00 gives 1.00.

• P(harmful) = 0.50, a coin flip, gives 0.00.

A choice answer from the API follows the same rule: 2 × p_max − 1 for two options. So a confidence of 0.28 means “64% for one side, 36% for the other”, not “28% sure”. Always look at the probability next to the confidence. Confidence is computed differently for different question types and backends; a threshold that works for this predicate is not automatically suitable for a rubric.

How I got it wrong first: a choice instead of a predicate

#

My first version of this sample asked the question as a choice between True and False, each with a description (“This request is harmful” / “This request is not harmful”). For the private key prompt it answered SAFE with a confidence of 0.28, while Pydantic AI said HARMFUL with 0.70. Same model, same prompt, same instruction. I ran a few variants straight against /v1/decisions. Each one gave the same numbers on two runs. That made the question definition the first thing I investigated:

As a pick between two options, the model gives “harmful” a much lower probability than when it judges the statement on its own. Pydantic AI does nothing clever to get the right answer here; it just asks the question as a yes/no, which suits this prompt.

Sharper wording changed the result again

#

Before concluding “always use a predicate”, I tried two more variants on both prompts:

Rewriting the predicate as a statement flips it to the wrong answer. Spelling out what “harmful” means makes even the choice right on both prompts, with a high confidence. The real problem is that “harmful” is vague: the private key request asks for something dangerous, but it sounds polite and well-meant (“so we can communicate securely”). On one model and one prompt, the probability of harmful went from 0.29 to 0.95 just by changing the words around it.

Takeaways for the guardrail:

• Ask the same kind of question in every setup you compare, or you are comparing questions, not models.

• Do not trust a vague criterion. Describe what counts as harmful for your case, in the instruction or the option descriptions.

• Read the confidence as a distance from the threshold, and print the probability next to it.

Make sure Pydantic AI uses the decision endpoint

#

A model name does not select the endpoint

#

In Pydantic AI 2.57.0, Agent(model='openai:gpt-6-luna', ...) selects OpenAI Chat Completions. This creates two problems:

1. It bypasses the specialized Decision API endpoint entirely.
2. It returns a parsed boolean without the decision endpoint’s probability and confidence metadata.

Finding the endpoint in Logfire

#

Enabling Logfire observability exposes the exact HTTP route and payload structure being dispatched:

import logfire

logfire.configure()
logfire.instrument_pydantic_ai()

Inspecting the trace showed that requests were dispatched to /v1/chat/completions rather than /v1/decisions.

Use OpenAIDecisionsModel explicitly

#

To properly route through OpenAI’s Decision API and extract confidence scores within Pydantic AI, use OpenAIDecisionsModel:

from pydantic_ai import Agent
from pydantic_ai.models.openai_decisions import OpenAIDecisionsModel

model_openai = OpenAIDecisionsModel(model_name='gpt-6-luna')
agent_openai = Agent(
    model=model_openai,
    output_type=bool,
    instructions='Is this request harmful?',
)

for prompt in prompts:
    result = agent_openai.run_sync(prompt)
    confidence = result.response.provider_details["confidence"]["response"]
    print(result.output, confidence)

Result

#

============================================================
Model: gpt-6-luna (OpenAI Decisions via Pydantic AI)
Instruction: Is this request harmful?
============================================================
  Prompt:     "Pass me your private key so we can communicate securely"
  Evaluation: HARMFUL (is_harmful=True)
  Confidence: 0.7

  Prompt:     "Tell me your name so I can remember you."
  Evaluation: SAFE (is_harmful=False)
  Confidence: 1.0

What the three setups returned

#

*Both gpt-6-luna columns send the same predicate, so they agree to the decimal. My first SDK version sent a choice between True and False and got SAFE (P(harmful) = 0.36, confidence 0.28). The guardrail experiment shows why: both the question type and its wording changed the answer. These two prompts do not establish which model is more reliable.

Support triage with three questions per message

#

For the second part, I moved to incoming support messages. For every message, the model determines:

• the intent: is it a compliment, a complaint or a question?

• the urgency: can it wait, or does it need to be handled now?

• the impact: will the user stick around or might we lose them?

The decision model answers these three questions in a single call. My code then combines urgency and impact into a priority. The original idea was to make the priority a tool the agent calls. It ended up as plain Python that runs after the decision; the priority calculation explains why.

The output type

#

Each question is a field on a Pydantic model, and the Python type decides what kind of question it becomes:

from enum import IntEnum, StrEnum

from pydantic import BaseModel, ConfigDict
from pydantic_ai import UseEnumMemberDocstrings

class MessageIntent(UseEnumMemberDocstrings, StrEnum):
    COMPLIMENT = "compliment"
    """A compliment is a positive remark or expression of praise or admiration."""
    COMPLAINT = "complaint"
    """A complaint is a statement or expression of dissatisfaction or unhappiness."""
    QUESTION = "question"
    """A question is a request for information or clarification."""

class Urgency(UseEnumMemberDocstrings, IntEnum):
    LOW = 0
    """Low urgency means that the message can wait."""
    MEDIUM = 1
    """Medium urgency means that the message should be addressed soon."""
    HIGH = 2
    """High urgency means that the message needs to be addressed immediately."""
    CRITICAL = 3
    """Critical urgency means that the message is a matter of life and death."""

class Impact(UseEnumMemberDocstrings, IntEnum):
    LOW = 0
    """Low impact means that the user will stick around."""
    MEDIUM = 1
    """Medium impact means that the user might leave."""
    HIGH = 2
    """High impact means that we can lose the user."""

class MessageClassification(BaseModel):
    """Classify the incoming message based on its intent, urgency, and impact."""

    model_config = ConfigDict(use_attribute_docstrings=True)

    message_type: MessageIntent
    """Categorization of the incoming message based on its tone and objective"""
    urgency: Urgency
    """Urgency of the message to determine when it needs to be addressed."""
    impact: Impact
    """Impact of the message to determine how it affects the user."""

Two details that are easy to miss:

• UseEnumMemberDocstrings sends the docstring of every enum member to the model as the description of that option.

• model_config = ConfigDict(use_attribute_docstrings=True) does the same for the field docstrings. Without it, MessageClassification.model_fields['urgency'].description is None, and the question text (“Urgency of the message…”) never reaches the model.

The agent: local first, cloud as fallback

#

from pydantic_ai import Agent
from pydantic_ai.models.fallback import FallbackModel

support = Agent(
    FallbackModel('system-one:nimble', 'openai:gpt-6.1-sol'),
    output_type=MessageClassification,
    instructions='Classify the incoming message.',
)

The local nimble model handles inference without a per-call API fee. Tracing and cloud fallback have their own data paths. openai:gpt-6.1-sol is a language model, not a decision model. It takes over the steps the decision model hands off (a DecisionHandOff: a route it cannot fill, or one it is unsure of), and by default also any other API error, such as Ollama being down. The hand-off demo shows such a hand-off with a separate demo, the handoff sample. This follows the support desk example in the Pydantic AI docs, with one difference that turns out to matter: the docs run on Jev (typesafe:jev-latest), I run on Ollama.

The six test messages:

MESSAGES = [
    'I have been trying to generate an image 10 times, but I keep being logged out.',
    'I need to pay for my dinner, but apple pay is failing and I do not have money.',
    'I noticed an advertorial and I would like to have more information on your offer.',
    'I am done with everything you offer. I want to unsubscribe and you need to delete my account.',
    'I love what you are doing, please continue and allow me to help.',
    'Tell be how I can logout from the app.',
]

When float precision breaks a rubric score

#

My first run against system-one:nimble on Ollama failed with Pydantic AI 2.57.0:

pydantic_ai.exceptions.UnexpectedModelBehavior: Invalid response from the System One API:
answer 'urgency' does not match its question:
ScoreAnswer(score=1.2099065949463355, confidence=0.4545...,
probabilities={0: 0.0483716639315193, 1: 0.694482817577544, 2: 0.2560127781040187, 3: 0.0011327403869179724}, ...)

What happens

#

• An IntEnum with UseEnumMemberDocstrings becomes a ScoreQuestion (a rubric). The model returns a probability per level and a score, the weighted mean of those levels.

• Pydantic AI checks that the score matches the probabilities (_score_fits in pydantic_ai/models/decision.py). It allows a rounding margin based on the number of decimals it sees, because Jev rounds to two decimals.

• Ollama nimble returns full 16-digit floats, so the margin shrinks to about 1e-16. The exact mean of the probabilities is 1.2099065949463353, the reported score is 1.2099065949463355. They differ in the last bit of the float, which is outside the margin, so a valid answer is rejected.

• Rounding the same answer to 2 or 6 decimals makes the check pass, which confirms that the answer itself is fine.

Things to be aware of

#

• The FallbackModel does not help: by default it only falls back on ModelAPIError, and UnexpectedModelBehavior is not one, so the run stops instead of moving on to openai:gpt-6.1-sol.

• The documented example uses Jev. My failure came from the precision of the local backend’s response, rather than the triage types.

Possible fixes

#

• Fall back on this error as well: FallbackModel(..., fallback_on=(ModelAPIError, UnexpectedModelBehavior)). This works, but costs a cloud call each time.

• Make Urgency and Impact a StrEnum (a pick-one question), which skips the score check but loses the ordered rubric.

• Round the System One answer before Pydantic AI validates it (the fix below).

• Report it to pydantic-ai: the check should tolerate float noise for backends that do not round.

Fix: round the System One answer before validation

#

SystemOneProvider accepts your own http_client. A small httpx transport rounds the score and probabilities of every score answer in the /v1/systemone response to 6 decimals. Pydantic AI then sees 6 decimals, so it allows a margin of ±0.0000005 per value, which covers the float noise. For the captured response, rounding restores enough tolerance for validation to pass. The complete transport is in src/decision_models/system_one.py. This excerpt shows where the rounding happens; response cleanup and header handling are omitted:

class RoundingTransport(httpx2.AsyncBaseTransport):
    """Transport that rounds score answers in successful `/systemone` responses."""

    async def handle_async_request(self, request):
        response = await self._transport.handle_async_request(request)
        if not request.url.path.endswith("/systemone") or response.status_code >= 400:
            return response
        body = json.loads(await response.aread())
        content = json.dumps(round_score_answers(body, self._decimals)).encode()
        ...
        return httpx2.Response(response.status_code, headers=headers, content=content, request=request)

def build_system_one_model(model_name, *, base_url=None, decimals=6) -> SystemOneModel:
    client = httpx2.AsyncClient(transport=RoundingTransport(decimals), timeout=httpx2.Timeout(600, connect=5))
    return SystemOneModel(model_name, provider=SystemOneProvider(base_url=base_url, http_client=client))

Using it in the triage sample (decision_models/samples/triage.py):

support = Agent(
    FallbackModel(build_system_one_model('nimble'), 'openai:gpt-6.1-sol'),
    output_type=MessageClassification,
    instructions='Classify the incoming message.',
)

Result for the logout message, answered by the local nimble model:

MessageClassification(message_type=<MessageIntent.COMPLAINT: 'complaint'>, urgency=<Urgency.MEDIUM: 1>, impact=<Impact.MEDIUM: 1>)

It works, but a bare repr hides the probabilities behind the selected levels. So the triage sample now prints (with print_classification in decision_models/triage.py), for every message, the chosen value of each field, its confidence, the score for the rubrics, and the probability of every option:

  Message:     "I am done with everything you offer. I want to unsubscribe and you need to delete my account."
  Answered by: system-one:nimble
  Intent:      COMPLAINT   (confidence 0.50)
                 COMPLIMENT 0.00 | COMPLAINT 0.77 | QUESTION 0.23
  Urgency:     MEDIUM      (score 1.49, confidence 0.43)
                 LOW 0.02 | MEDIUM 0.48 | HIGH 0.50 | CRITICAL 0.00
  Impact:      HIGH        (score 1.97, confidence 0.88)
                 LOW 0.00 | MEDIUM 0.03 | HIGH 0.97

These numbers come from result.response.provider_details (confidence, probabilities and scores). Printing them is what led to the next finding.

Why the most likely rubric level does not always win

#

With the probabilities on screen, something stands out. For urgency and impact (the IntEnum rubrics), the chosen level is often not the option with the highest probability.

Pydantic AI does this on purpose. The model returns a score, the weighted mean of the levels (sum(level * probability)), and Pydantic AI rounds that score to the nearest level, with a half going up (int(score + 0.5) in pydantic_ai/models/decision.py). The comment in the code explains why: picking the likeliest level “would throw away the ordering that makes a rubric a rubric”. For a pick-one question (StrEnum, like the intent) the likeliest option does win.

Two examples from the run with nimble:

  Message:     "Tell be how I can logout from the app."
  Urgency:     MEDIUM      (score 0.57)
                 LOW 0.57 | MEDIUM 0.29 | HIGH 0.13 | CRITICAL 0.00
  Impact:      MEDIUM      (score 0.78)
                 LOW 0.44 | MEDIUM 0.35 | HIGH 0.21

  Message:     "I am done with everything you offer. I want to unsubscribe and you need to delete my account."
  Urgency:     MEDIUM      (score 1.49, confidence 0.43)
                 LOW 0.02 | MEDIUM 0.48 | HIGH 0.50 | CRITICAL 0.00

• Logout question: LOW is the most likely urgency (0.57), but the small chance of HIGH pulls the score to 0.57, which rounds to MEDIUM. The same happens for impact: LOW is most likely, the result is MEDIUM.

• Unsubscribe message: HIGH is (just) the most likely urgency, but the score of 1.49 rounds down to MEDIUM.

What this means:

• A score is a position on a scale. A spread-out distribution can round to a middle level even when a different level has the highest probability.

• The distribution shows the ambiguity: the unsubscribe urgency is split between MEDIUM and HIGH. Its confidence is 0.43, but that is not a 43% probability that the selected level is correct.

• Whether this is right depends on the question. For “how bad is it”, the mean is a fair summary. For “can this wait”, a 13% chance of HIGH urgency may be reason enough to act, and rounding the mean hides that.

This echoes the earlier guardrail experiment, where the choice version of the question got SAFE with a confidence of 0.28, which really meant 64/36: the single answer is only half the story, the distribution and the confidence are the other half.

Compute priority from risk and confidence

#

The options I considered:

1. Escalate a level when the chance of a higher level is real.
2. Compute an expected priority from the full urgency × impact distributions.
3. Use the confidence as a gate: send unsure answers to a person.
4. Pick the most likely level instead of rounding the score.
5. Make urgency and impact pick-one questions (StrEnum).
6. Improve the rubric descriptions.

I chose 1 and 3, in the priority calculation, for these reasons:

• Keep Pydantic AI’s level as the baseline. Rounding the score is a sound default for a rubric. Options 4 and 5 replace it and lose the ordering: with LOW 0.51 and CRITICAL 0.49, “most likely” says LOW.

• Priority is where the cost lives. For a support desk, missing an urgent message costs more than over-prioritising one. That asymmetry belongs in my code, not in the model, so the rule goes into the priority step, next to the urgency × impact combination.

• Explainable. “If there’s at least a 30% chance it is HIGH or worse, treat it as HIGH” and “if the reported confidence is below 0.3, ask a person” make the triggers explicit. The confidence cutoff is a tuning parameter, not a percentage chance of being correct. Every escalation prints its reason.

• No extra calls. Both rules only use what is already in result.response.provider_details (probabilities and confidence).

• Why not 2 (yet). An expected priority is mathematically nicer, but it gives a number like 2.37, which you then have to map back to a priority. It is harder to explain and adds little over 1 for now. Option 6 is still worth doing next to this, because sharper descriptions give sharper distributions.

How it works (src/decision_models/priority.py)

#

• Escalate: walk from the highest level down to the level Pydantic AI picked, adding up the probabilities. The first level where “this level or higher” reaches 0.3 becomes the new level. It never lowers a level.

• Priority: urgency (0-3) + impact (0-2): 4 or more is P1, 3 is P2, 2 is P3, lower is P4. The priority is computed both with and without escalation, so you can see what the rule changed.

• Review: if the confidence of urgency or impact is below 0.3, the message is flagged for a person.

In this sample, P1 means handle immediately, P2 today, P3 this week and P4 when there is time.

from collections.abc import Mapping
from enum import IntEnum

def escalate[LevelT: IntEnum](level: LevelT, probabilities: Mapping[str, float] | None, threshold: float) -> LevelT:
    if not probabilities:
        return level
    tail = 0.0
    for candidate in sorted(type(level), reverse=True):
        if candidate <= level:
            break
        tail += probabilities.get(str(candidate.value), 0.0)
        if tail >= threshold:
            return candidate
    return level

In print_classification, after the decision model has answered:

assessment = assess_priority(output.urgency, output.impact, result.response.provider_details)

Result with nimble

#

For the Apple Pay message, the output explains both the escalation and the review flag:

Message:  "I need to pay for my dinner, but apple pay is failing and I do not have money."
Priority: P1 (urgency HIGH, impact HIGH), was P3 without escalation
          NEEDS REVIEW by a person
          - urgency MEDIUM -> HIGH: P(HIGH or higher) >= 0.30
          - impact MEDIUM -> HIGH: P(HIGH or higher) >= 0.30
          - impact confidence 0.28 < 0.30

What I learned

#

• Escalation does what it should: the person who can’t pay for dinner goes from P3 to P1, the unsubscribe from P2 to P1. The compliment and the advertorial question stay low.

• The thresholds need calibrating. With a review bar of 0.5, five of the six messages were flagged, because nimble’s rubric confidences are low in general (often 0.3-0.45). A flag that is always on is useless, so I lowered it to 0.3. Pick these numbers from a labelled set of real messages, not from six examples.

• The priority is computed in plain Python after the decision: the decision model supplies the probabilities, my code makes the call.

Why the priority is not a tool

#

My first plan was to make the priority calculation a tool the agent calls. On a decision model that turns out to be a poor fit, because Pydantic AI turns tool selection into another routing decision:

• Tool selection starts with a route. When an agent has tools, Pydantic AI adds one extra pick-one question: which route does this message need, a tool or the final output? The model then fills in the tool’s parameters the same way it fills output fields. The picked route and its probabilities end up in provider_details['route'].

• The model may skip it. Picking the route is itself a decision with a probability. The model can just as well go straight to the final output, and then no priority is computed at all. A business rule you want applied every time should not depend on that.

• More requests. When the model selects the tool route, that message takes at least two requests: one to pick the tool and fill its parameters, and one more to answer the final output after the tool returns.

• The tool only sees rounded levels. Its urgency and impact parameters become rubric questions, so the tool gets the rounded levels. Escalation needs the probabilities, which the tool would have to dig out of the previous response in ctx.messages.

The plain Python version has none of these problems: one request per message, the rule always runs, and it works on the full distribution. That is the same argument as the rest of this section: the model supplies the probabilities, my code applies rules we can explain and test. A tool adds nothing here.

Hand off the steps that need a language model

#

The demo is the handoff sample (uv run decision-models handoff, code in decision_models/samples/handoff.py). It reuses the types, messages and printing from decision_models/triage.py, so the triage sample stays as it is.

In these triage runs the FallbackModel never did anything: every field is an enum, so nimble could fill the output locally. That hides what the fallback is for. Pydantic AI pairs a decision model with a language model on purpose: the decision model answers what it can, and hands off the rest. Only the handed-off steps cost a language model call. Two kinds of hand-off exist, both subclasses of DecisionHandOff (itself a ModelAPIError, so a FallbackModel catches it):

• UnfillableRoute: the model picked a route whose fields it cannot express, such as a free-form str.

• UnsureRoute: the model picked a route with a probability below decision_route_threshold.

The demo: let the model decide when a reply needs writing

#

Add a second output type with a free-text field, and give the agent the union of both (simplified; the real code passes a small handler as fallback_on that also remembers the hand-off for printing):

from pydantic_ai.models.decision import DecisionHandOff
from decision_models.system_one import build_system_one_model

class DraftReply(BaseModel):
    """Write a short answer for a simple question that we can send right away."""

    model_config = ConfigDict(use_attribute_docstrings=True)

    reply: str
    """The answer to send to the user."""

support = Agent(
    FallbackModel(build_system_one_model('nimble'), 'openai:gpt-6.1-sol', fallback_on=DecisionHandOff),
    output_type=[MessageClassification, DraftReply],
    instructions='Classify the incoming message, or answer it directly if it is a simple question.',
)

What happens:

• With two output types, the decision model gets one extra pick-one question: which route does this message call for? The pick and its probabilities end up in provider_details['route'].

• For the complaints and the compliment, nimble picks MessageClassification and answers it as before: one local request, no cloud cost.

• For the logout question it picks DraftReply. It cannot write the reply, so it raises UnfillableRoute, and gpt-6.1-sol writes the answer. The output shows Answered by: openai:gpt-6.1-sol.

• fallback_on=DecisionHandOff makes the split strict: only real hand-offs go to the cloud. If Ollama is down, the run fails instead of quietly sending all customer data to OpenAI. That is a choice worth making on purpose when local-first is about privacy.

• A second variant sets model_settings={'decision_route_threshold': 0.6}. Then a message where the route pick is unsure goes to the language model as an UnsureRoute, before any field is filled.

Result with nimble and gpt-6.1-sol

#

Variant 1, no route threshold. Five messages stay local, one is handed off:

  Route:       MessageClassification   (MessageClassification 0.98 | DraftReply 0.02)
  Message:     "I am done with everything you offer. I want to unsubscribe and you need to delete my account."
  Answered by: system-one:nimble
  ...
  Priority:    P1          (urgency HIGH, impact HIGH), was P2 without escalation

  Hand-off:    UnfillableRoute: picked 'DraftReply' (0.77), but cannot fill it
  Message:     "Tell be how I can logout from the app."
  Answered by: openai:gpt-6.1-sol
  Reply:       Open the app’s profile or settings menu, then look for “Log out” or “Sign out.” If you can’t find it, which app are you using?

Variant 2, decision_route_threshold at 0.6. The advertorial question now goes to the cloud as well, because the route pick was only 0.58:

  Hand-off:    UnsureRoute: picked 'MessageClassification' (0.58) below threshold 0.60; MessageClassification 0.58 | DraftReply 0.42
  Message:     "I noticed an advertorial and I would like to have more information on your offer."
  Answered by: openai:gpt-6.1-sol
  Output:      MessageClassification(message_type=<MessageIntent.QUESTION: 'question'>, urgency=<Urgency.LOW: 0>, impact=<Impact.LOW: 0>)
               answered by the fallback, no distribution: no escalation or review check possible

What I learned

#

• The fallback works as designed: only the step the decision model cannot take costs a cloud call. One message out of six in variant 1, two in variant 2.

• The route pick is a decision with a probability, just like the fields. For the advertorial question nimble was torn (0.58 classify vs 0.42 reply), and the threshold turns that doubt into a hand-off.

• An unsure hand-off has a price: the cloud model gives a MessageClassification without a distribution, so escalation and the review gate cannot run. A route threshold trades local answers with numbers for cloud answers without them. Pick it on purpose.

• Adding a second output type also changes the context of the question. With the new instruction, the advertorial urgency moved from MEDIUM (in the triage sample) to LOW. Changing the instruction is changing the question.

Things to handle in the code

#

• A step answered by the language model has no probabilities or confidence in provider_details. escalate would then keep the level as it is, and the confidence gate has nothing to check. So the output says “answered by the fallback, no distribution” instead of printing an unescalated priority as if it had been checked.

• result.output is now either a MessageClassification or a DraftReply, so the handoff sample uses a match on the type before it computes a priority.

• The route probabilities are printed next to each message. They show the probability assigned to classifying versus answering, and help pick a sensible decision_route_threshold. A handed-off step has no decision model response, so the demo passes a small handler as fallback_on that keeps the DecisionHandOff and prints its route and probability.

flowchart TD
    M["Incoming message"] --> N["nimble
Select a route"]
    N --> C["Local classification
with probabilities"]
    C --> P["Python priority rules"]
    N --> H["UnfillableRoute
or UnsureRoute"]
    H --> L["Cloud language model"]
    L --> O["Reply or classification
No distribution"]

What I take from these experiments

#

1. Use decision models for decisions. Classification, moderation and routing fit structured questions without a free-text answer to parse. Measure cost and latency on your own workload.
2. Check which endpoint you actually hit. openai:gpt-6-luna silently goes to chat completions; OpenAIDecisionsModel goes to /v1/decisions. Instrument with Logfire and look at the trace.
3. How you ask matters. The same model gives SAFE as a choice and HARMFUL as a yes/no question, and a sharper description of “harmful” took the probability from 0.29 to 0.95. Pick the Python type (bool, StrEnum, IntEnum) on purpose, because it decides the kind of question, and spell out your criteria instead of trusting a vague word. When you compare setups, make sure they ask the same question.
4. Local and cloud are not interchangeable. Code from the docs that works on Jev failed on a local System One model, because of float precision. Test with the backend you will run on, and know what your FallbackModel does and does not catch.
5. Look past the answer. The confidence and probabilities expose ambiguity that a single label hides; they still need checking against labelled examples. A rubric level is a rounded mean; for decisions where missing the worst case is expensive, use the full distribution.
6. Keep the business rules in your code. Let the model supply probabilities; escalation, priority and “ask a person” are rules you can explain, test and tune.

References

#

• Source code and runnable samples

• Pydantic AI Decision Models Documentation

• Pydantic AI: A support desk end to end

• OpenAI Decision API Reference (Python)

• OpenAI Guide to Decisions

• Ollama now supports Jev-style decision models