In the first article of this series, I ran the same whisky-tasting agent with ordinary tool calls and with Code Mode in Monty. Plain tool calls were cheaper on a small task; Code Mode won once the agent had to work through a catalogue of 504 whiskies. After publishing it, I got a question I had already been asking myself: why choose for the agent? What happens if you give it both routes and let it decide per step?
It sounds like the best of both worlds. A lookup is one tool call; a loop over 21 catalogue pages is a short program. An agent that picks the right route at the right moment should beat both fixed variants. But a choice is also one more decision the model can get wrong, and the extra route costs tokens in every request, whether the model uses it or not.
So I built a hybrid mode for the same two scenarios and ran it with two models: gpt-6.1-sol and
the smaller gpt-6-luna. Every plan is still checked by the independent validator from part 1,
and every run now also records which route the model took at each step. You will see that the
data size decides the route far more than the option does, that the stronger model found a good
split between lookups and code without being told, and that the one mistake I saw was the budget
trap from part 1 in a new form. At the end, I add a small calculator tool and measure what it
fixes and what it costs.
Run the samples#
All examples are in my whisky-sandbox-demo repository.
The versions are the same as in part 1: Python 3.12, uv, pydantic-ai-slim[openai] 2.68.0 and
pydantic-ai-harness[codemode,smol] 0.68.0 with Monty 1.6.0.
git clone https://github.com/jettro/whisky-sandbox-demo.git
cd whisky-sandbox-demo
uv sync
cp .env.example .env # set OPENAI_API_KEY and DEMO_MODEL, e.g. gpt-6.1-solThe new samples have their own Makefile targets:
make demo-hybrid # scenario 1: the agent chooses per step
make season-hybrid RUNS=3 # scenario 2: the agent chooses per step
make season-hybrid-calc RUNS=3 # scenario 2, plus the price_season calculator
make check # ruff and 88 offline testsThe new code sits next to the existing demos:
src/whisky_sandbox_demo/
hybrid.py # the run_code toolset, the neutral instruction and the route trace
season/
calculator.py # price_season, the fix experiment
demos/
hybrid.py, season_hybrid.py, season_hybrid_calc.pyOffering the same tools twice#
Implementation#
In part 1, the CodeMode capability took over the agent’s tools: they disappeared as tool calls
and came back as async functions inside run_code. For the hybrid mode, I wanted both at once.
Reading the harness source, I found that CodeMode cannot do that by itself. Its tools selector
splits the toolset: each tool is either sandboxed or native, never both.
The way around it is to leave the agent’s tools alone and add a separate CodeModeToolset that
wraps its own copy of the same functions. From hybrid.py:
def code_toolset(tools, *, max_tool_calls, resource_limits) -> CodeModeToolset:
"""A `run_code` tool whose sandbox can call `tools`; the agent keeps its own direct copies."""
return CodeModeToolset(
wrapped=FunctionToolset(list(tools)),
max_tool_calls=max_tool_calls,
resource_limits=resource_limits,
)The agent gets the domain tools as before, plus this toolset:
run_code = code_toolset(
DOMAIN_TOOLS,
max_tool_calls=50,
resource_limits={"max_duration_seconds": 10, "max_memory": 128 * 1024 * 1024},
)
Agent(
model,
instructions=f"{INSTRUCTIONS}\n{HYBRID_INSTRUCTIONS}",
output_type=TastingPlan,
tools=list(DOMAIN_TOOLS),
toolsets=[run_code],
)The model now sees five tools: get_tasting_brief, search_whiskies, get_whisky_details,
get_supplier_offers and run_code. Inside run_code, the same four functions are callable
under the same names. The CodeModeToolset only exposes run_code at the top level, so the
names never clash.
flowchart LR
M["Model"] -->|"direct tool call"| T["Domain tools
(host functions)"]
M -->|"run_code(python)"| R["Monty sandbox"]
R -->|"await get_supplier_offers(...)"| T
The model also needs to know that both routes exist. I kept that instruction deliberately neutral:
HYBRID_INSTRUCTIONS = """\
You can use the catalogue tools in two ways: call them directly as tools, or call the same
functions from Python inside run_code. Choose whichever fits each step; you may mix both.
"""It says nothing about when to prefer either route. That one extra paragraph does make the hybrid a prompt variant, so I compare it to the earlier variants as an experiment, not as a controlled comparison.
Recording the route#
To see what the model chose, path_trace walks over the model responses and labels each one:
direct for domain tool calls, code for run_code, mixed for both in one response, and
output for the final answer. Each run saves this as path.json, together with a heuristic flag
that tells whether any snippet did arithmetic on prices or totals. A route looks like this:
direct > direct > code > code > outputScenario 1: one tasting, a small catalogue#
The first scenario is the blind tasting from part 1: four single malts for eight people, a €120 budget, one supplier and 20 real whiskies. I ran the hybrid three times with each model.
Result#
| gpt-6-luna | gpt-6.1-sol | |
|---|---|---|
| Valid plans | 3 / 3 | 3 / 3 |
Runs that used run_code | 0 / 3 | 3 / 3 |
| Typical route | direct > direct > direct > output | direct > direct > code > code > output |
| Model requests | 4–5 | 5–7 |
| Input tokens | 15.0k–22.8k | 21.6k–33.8k |
| Part 1, one run: tool calls / Monty | 8.8k / 29.6k | 8.9k / 22.5k |
The smaller model never touched run_code. It batched its lookups as direct tool calls, exactly
like the plain tool-calling variant, and returned a valid plan. But it paid for an option it never
used: the run_code description adds about 1.5k input tokens to every request, about 6k per run.
The larger model made the same switch in all three runs. It fetched the brief and searched the
catalogue with direct calls, and then moved to code for two things: fetching details and offers
from three suppliers in one asyncio.gather, and checking its choice. This is its second snippet
from the first run:
import math
selected_ids = ['W01','W02','W09','W10']
details = {w['id']:w for w in results[0]['whiskies']}
offers = {o['whisky_id']:o for o in results[1]['offers']}
checks = []
for wid in selected_ids:
w = details[wid]
o = offers[wid]
n = math.ceil(8*20/w['bottle_ml'])
checks.append({'id':wid,'bottles':n,'line_total_cents':n*o['price_cents'],
'stock_bottles':o['stock_bottles'],
'valid':w['type']=='single_malt' and w['abv']<=50 and o['in_stock'] and o['stock_bottles']>=n})
total = sum(c['line_total_cents'] for c in checks)
{'checks':checks,'total_cents':total,'remaining_cents':12000-total,'within_budget':total<=12000}The four ids are hard-coded: the model chose the whiskies itself and used code to check bottles, line totals and the budget. That is the split I hoped for: look things up directly, and compute in code. The model found it without any hint. I trimmed a few fields from the snippet for width; the full version is in the repository output.
What I learned#
- On a small task, the option mostly adds weight. The smaller model ignored it and paid the token cost anyway. The larger model used it well, but ended up at about the cost of Monty alone, still well above plain tool calls.
- A good split exists and a strong model can find it. Direct calls for discovery, code for the fan-out and the arithmetic. I think that is the strongest argument for the hybrid: you do not have to design the split yourself.
Scenario 2: a club season over the full catalogue#
The second scenario is where Code Mode won in part 1: plan four themed evenings with four whiskies
each for 30 members, from a paginated catalogue of 504 whiskies across three suppliers, within a
€540 season budget, shipping included. I ran the hybrid three times with each model, and five more
times with gpt-6-luna to get a firmer view of how often it fails.
Result#
| gpt-6-luna | gpt-6.1-sol | Part 1 with gpt-6.1-sol: tool calls / Monty | |
|---|---|---|---|
| Runs | 8 | 3 | 3 each |
| Valid seasons | 7 / 8 | 3 / 3 | all valid |
Runs that used run_code | 8 / 8 | 3 / 3 | – |
| Model requests, median | 10 (6–19) | 7 | 8 / 8 |
| Input tokens, median | 156k (39k–390k) | 67k (60k–88k) | 208k / 125k |
| Largest single request, median | 25.6k | 17.1k | 48.6k / 27.8k |
With a large catalogue, both models moved to code right after get_season_brief. So the smaller
model can choose code; it just does not when the problem is small. Every gpt-6.1-sol run
followed the same route:
direct > code > code > code > code > code > outputIts first snippet paged through all 21 catalogue pages in one asyncio.gather, computed from the
product_count in the brief it had just fetched directly. With a median of 67k input tokens, the
hybrid used about half of Monty’s 125k from part 1. I think the direct first step helps: the brief
is already in the context, so the first snippet can start fetching instead of spending a round
trip on discovery. Treat that as a hint, not a result. These are three runs, and in part 1 a
single run pointed the wrong way.
The smaller model was far less consistent. For the same task, it used between 6 and 19 requests
and between 39k and 390k input tokens. Some runs went back to direct get_products calls after
several snippets, others stayed in code until the end. The choice of route is one more degree of
freedom, and a weaker model uses it differently every time.
How it got it wrong: the budget trap in a new form#
One of the eight gpt-6-luna runs returned an invalid season. The validator reported:
FAIL order S1: planned 53020+695=53715, expected 53020+0=53020
FAIL season_total: 53715 vs 53020The model charged €6.95 shipping on a De Helm order of €530.20. De Helm ships free above €100, so the season total was €537.15 where it should have been €530.20. The run had used code eight times, to fetch, filter and compare. None of those snippets mentioned shipping: the model wrote the orders and the season total itself, in its final answer. Its own explanation even mentions the free-shipping threshold, and still the number is wrong.
This is the trap I described in part 1: the budget is the rule where an agent has to do exact arithmetic, and a model doing sums in its head gets it wrong from time to time while sounding confident. Code Mode was supposed to move that arithmetic into a program. In the hybrid, the model decides which arithmetic goes into code, and here it left out the one sum that made the plan invalid.
My heuristic flag did not help either. It reported “arithmetic on prices in code” for this run, because earlier snippets compared prices. Doing some arithmetic in code is not the same as doing all of it there.
The fix: a calculator tool#
If the risk is in the final numbers, the obvious fix is not more freedom but less: make the
arithmetic a tool. price_season in season/calculator.py takes the selection and returns every
number the plan needs.
Implementation#
class EveningSelection(BaseModel):
theme: Theme
supplier_id: str
product_ids: list[str] = Field(description="In pouring order, lightest first.")
def price_season(evenings: list[EveningSelection]) -> SeasonQuote:
"""Price a candidate season exactly: bottles, line totals, evening subtotals, one order per
supplier with its shipping (free above the threshold), the season total and the budget check.
Copy these numbers into the plan instead of computing them yourself. The quote does not check
eligibility, themes or order; it lists problems it can see (not offered, stock, repeats).
"""The calculator deliberately does not judge eligibility, themes or tasting order. Choosing the
whiskies stays with the model; only the arithmetic moves. It uses the same bottles_for and
shipping_cents helpers as the validator, and an offline test checks that its quote matches the
validated reference season exactly.
The variant offers price_season like the other tools, directly and inside run_code, with one
more instruction:
CALC_INSTRUCTIONS = """\
Before you return the plan, price your final selection with price_season and copy its numbers
(bottles, line totals, subtotals, orders with shipping, season total) into the plan.
"""Result#
| gpt-6-luna hybrid | gpt-6-luna + calculator | gpt-6.1-sol hybrid | gpt-6.1-sol + calculator | |
|---|---|---|---|---|
| Runs | 8 | 5 | 3 | 3 |
| Valid seasons | 7 / 8 | 5 / 5 | 3 / 3 | 3 / 3 |
| Model requests, median | 10 (6–19) | 14 (12–24) | 7 | 8 (8–9) |
| Input tokens, median | 156k (39k–390k) | 151k (133k–544k) | 67k (60k–88k) | 118k (105k–142k) |
Every run used the calculator. gpt-6-luna always called it as a direct tool, sometimes twice with
the same selection. gpt-6.1-sol called it from inside run_code in two runs and directly in
one:
selection=[{'theme':'speyside','supplier_id':'S1','product_ids':['H86067','H81795','H99140','H99838']},
{'theme':'islay','supplier_id':'S1','product_ids':['H102685','H986','H1388','H101807']},
...]
quote=await price_season(evenings=selection)
quoteAll eight seasons were valid, and every quote showed one De Helm order with free shipping.
What I learned#
- The fix removes an error class, not a measured failure rate. Without the calculator,
gpt-6-lunafailed once in eight runs. Five clean runs with the calculator do not prove that the rate went down; that would take far more runs. What changed is where the numbers come from: a function that shares its logic with the validator, instead of the model’s own arithmetic. - It is not free.
gpt-6.1-solneeded one more request and its median input went from 67k to 118k, partly because the full quote comes back into the context. Then again, 118k is about where Monty landed in part 1, so the 67k of the first batch may have been the lucky one. Three runs per variant are not enough to tell. - The weaker model stays erratic. With or without the calculator,
gpt-6-lunavaried by a factor of four or more in tokens for the same task.
Things to be aware of#
- The hybrid is a prompt variant. It adds one neutral paragraph about the two routes, and the calculator variant adds another. I compare them to part 1 as experiments.
- Small numbers of runs. Three to eight runs per variant, two models, one snapshot of one
shop. The routes were remarkably consistent for
gpt-6.1-sol; the token numbers were not. - You pay for the option in every request. The
run_codedescription is sent with every request, also by a model that never uses it. - Wiring both routes is a workaround.
CodeMode’s tool selector does not offer “both”, so the hybrid uses a second toolset with copies of the same functions. Tools with side effects would then be reachable twice; in this demo, all tools only look things up.
What I take from these experiments#
- The data decides the route, not the option. On a small task, the smaller model never wrote code and the larger one wrote just a little; on a 504-whisky catalogue, both moved to code immediately.
- A strong model finds a good split by itself.
gpt-6.1-solused direct calls for discovery and code for fan-out and arithmetic, in every run, without being told how. - Giving a choice adds variance. The smaller model took a different route almost every run, with a tenfold spread in tokens. If you need predictable cost, a fixed mode is easier to reason about.
- Code Mode only protects the arithmetic the agent puts in code. The one invalid season was a shipping sum the model wrote itself, after eight snippets of code.
- Make exact numbers a tool, and validate them anyway. A small calculator that shares logic with the validator removes the error class; the validator still catches whatever is left.




