↓ Skip to main content
  1. Blog/

Give your agent a sandbox: tool calls, Monty and Smol compared

Jettro Coenradie
Author
Jettro Coenradie
Software architect and search enthusiast. I write about AI, search, cloud, and software development.

An agent that can only call tools works one step at a time. It asks for some data, reads the result, decides on the next call, and every result lands in its context, whether it needs all of it or not. A popular answer is to let the agent write code instead: a short program that calls the tools, filters the results and returns only what matters. That code has to run somewhere safe, so the next question is which sandbox.

Pydantic AI Harness offers two very different ones. Monty is a small, sandboxed Python interpreter that runs inside your own process; with the CodeMode capability, the agent’s tools become async functions it can call from the code it writes. Smol gives the agent a microVM on your machine, with its own kernel and filesystem, where it can write files and run full Python programs with third-party libraries.

I wanted to know when these sandboxes actually improve an agent, rather than assume they do. So I built one agent task, organising a whisky blind tasting with real whiskies from a Dutch shop, and ran it three ways: with ordinary tool calls, with Code Mode in Monty, and with Monty plus a Smol VM that produces a printable tasting pack. Every plan is checked by an independent validator, and every run records requests, tokens and tool calls.

The results surprised me. On the small task, Code Mode used two to three times more input tokens than plain tool calls. Only when I built a second scenario around a catalogue of 504 whiskies did Monty show what it is good at: about 40% fewer input tokens and a request size almost halved. You will see both scenarios, the code the model actually wrote, what went wrong along the way, and how Smol fits in when the agent needs to produce real files.

Run the samples
#

All examples are in my whisky-sandbox-demo repository.

The project uses Python 3.12, uv, pydantic-ai-slim[openai] 2.68.0 and pydantic-ai-harness[codemode,smol] 0.68.0, which brings Monty 1.6.0 and smolmachines 1.35.0. Note that the harness has no openai extra; the OpenAI dependency comes from pydantic-ai-slim. You clone the repository, install the dependencies and add your OpenAI key like this:

git clone https://github.com/jettro/whisky-sandbox-demo.git
cd whisky-sandbox-demo
uv sync
cp .env.example .env   # set OPENAI_API_KEY and DEMO_MODEL, e.g. gpt-6.1-sol

Logfire tracing is optional: set LOGFIRE_TOKEN to send traces, leave it empty to run without. The Makefile has a target for every sample:

make demo-tools          # scenario 1, ordinary tool calls
make demo-monty          # scenario 1, Code Mode in Monty
make demo-monty-program  # scenario 1, Monty with a "write one program" instruction
make guest-image         # once: prepare the libraries for the Smol VM
make demo-smol           # scenario 1 plus a printable tasting pack, built in a Smol VM
make season-compare RUNS=3   # scenario 2, both variants, three runs each
make check               # ruff and 83 offline tests

The code is a small package; each demo is a module with a main():

src/whisky_sandbox_demo/
    tools.py           # the four domain tools, identical in every demo
    models.py          # TastingPlan and the tool result types
    validation.py      # checks a plan against the data, independent of the agent
    metrics.py         # requests, tokens, top-level and nested tool calls
    guest.py           # everything about the Smol guest VM
    demos/
        tool_calls.py, monty.py, monty_program.py, monty_smol.py
        season_tool_calls.py, season_monty.py
    season/            # scenario 2: catalogue, tools, validation, runner
    data/              # the whisky and supplier snapshots

The scenario: a blind tasting with real whiskies
#

The task is the same for every variant:

Organise a blind tasting for eight people with four different single malt whiskies. The total bottle budget is €120. Begin accessible and sweet, and finish smoky. Each whisky must be at most 50% ABV. Buy all bottles from one supplier, with sufficient stock. Each participant receives 20 ml of each whisky.

The data is a snapshot of 20 real whiskies from Slijterij De Helm, taken on 15 November 2026: names, ABV, prices and stock. Please support the shop if you like a fine glass of whisky from time to time. You can always visit them and tell them how you got to know them. They do not know I am doing this, so be gentle with the endpoint. The shop runs WooCommerce, and its public Store API returns prices in cents and stock as text like “2 op voorraad”, which made it easy to keep only facts. Claude generates the tasting notes into short summaries. The flavour scores are also Claude’s illustrative estimates. Two extra suppliers are clearly fictional; they exist so that the The one supplier rule has something to trade off.

The data has a few traps built in, most of them for free:

  • The cask strength versions of the budget Islay malts are very much in demand but at 58% ABV.
  • The shop’s “malt” category contains blended malts like Monkey Shoulder and Rock Island, so an Agent that trusts category names selects a whisky which is not a single malt.
  • Some bottles are sold out or are backorder only.
  • Hart Brothers Islay Peated is precisely 50%, which tests “at most”.

Four tools that only look things up
#

The agent gets four domain tools: get_tasting_brief(), search_whiskies(style), get_whisky_details(ids) and get_supplier_offers(supplier_id, whisky_ids). They are plain Python functions with typed Pydantic return values, and they only look things up. Filtering on ABV and type, comparing suppliers, doing the arithmetic and choosing the order is the agent’s job.

def get_supplier_offers(supplier_id: str, whisky_ids: list[str]) -> SupplierOffersResult:
    """Return one supplier's price (euro cents) and stock (bottles) for several whiskies.

    Whiskies the supplier does not sell are listed in `not_carried_ids`.
    """

The tools take batches where that makes sense. I think splitting them artificially, one call per whisky, would make plain tool calling look worse than it is and turn the comparison into a setup.

Validate the plan, not the conversation
#

The agent returns a TastingPlan: the supplier, the four whiskies in order with bottles, unit prices and line totals, the total, and its reasoning. validation.py recomputes everything from the data: four distinct single malts, ABV at most 50%, one supplier that sells all four, enough stock, correct prices and arithmetic, and the budget. The flavour order is reported separately, as a recommendation, as “sweet to smoky” is a judgement rather than a constraint.

How I got it wrong first: a budget that never mattered
#

A constraint only tests an agent when it can be broken. The budget is the one rule in this task that forces a trade-off: the agent has to add up four prices, compare combinations and give up a favourite when the total runs over. That is exactly the arithmetic and search where code should beat a model doing sums in its head. If any choice fits the budget, the agent never has to do that work, and the validator cannot tell a careful agent from a careless one.

My first budget was €200. During the experiments, I noticed the model returning hundreds of combinations, and a brute-force count explained why: 13 whiskies qualify at De Helm, and every one of the C(13, 4) = 715 combinations of four fits under €200. The budget never influenced anything. Every variant would pass, whether it checked the total or not, so the comparison would show nothing.

At €120 only 7 sets fit, all at De Helm; the cheapest is €114.80. Richer whiskies in the middle of the flight, such as Glen Scotia Double Cask, no longer fit. Now a wrong sum or one greedy pick produces an invalid plan, and the validator catches it. The test suite checks that the budget keeps limiting the choice, so a later change to the data cannot quietly make it toothless again. I think this is the easiest trap to fall into when you design a task for comparing agents: a constraint that looks strict on paper but never decides anything.

Baseline: ordinary tool calling
#

Implementation
#

The first demos share the agent definition. Only the capabilities differ:

def build_planning_agent(capabilities=(), extra_instructions=None) -> Agent:
    instructions = INSTRUCTIONS if not extra_instructions else f"{INSTRUCTIONS}\n{extra_instructions}"
    return Agent(
        config.get_agent_model(),
        instructions=instructions,
        output_type=TastingPlan,
        tools=list(DOMAIN_TOOLS),
        capabilities=list(capabilities),
    )

You run the baseline with make demo-tools. The model calls the tools directly, and Pydantic AI lets it request several in one response.

Result
#

With gpt-6.1-sol, the run ended like this:

Validation: VALID (checked against fixtures)
  constraints passed: 40/40

Metrics (tool_calls, openai:gpt-6.1-sol)
  elapsed: 27.83s, model requests: 4
  tokens in/out: 8921/1449
  top-level tool calls: {'get_tasting_brief': 1, 'search_whiskies': 1, 'get_whisky_details': 1, 'get_supplier_offers': 3, 'final_result': 1}

Four model requests: the brief, the catalogue, then the details and all three suppliers’ offers in one response, and finally the plan. Plain tool calling already runs the independent lookups in parallel. That turned out to matter for everything that follows.

Code Mode with Monty
#

CodeMode folds the agent’s tools into a single run_code tool. The model writes Python that calls the tools as async functions, and Monty runs that Python inside the host process. Every call from the code is dispatched back to the real tool on the host; the tool implementations do not move anywhere.

flowchart LR
    subgraph TC["Ordinary tool calls"]
        M1["Model"] -->|"tool call"| T1["Domain tools
(host)"] T1 -->|"full result"| M1 end subgraph CM["Code Mode"] M2["Model"] -->|"run_code(Python)"| MO["Monty sandbox
(in the host process)"] MO -->|"await tool(...)"| T2["Domain tools
(host)"] T2 --> MO MO -->|"last expression only"| M2 end

Monty is a Python subset: standard library only (math, itertools, json, collections and a few more), no third-party imports, and no file, environment, network or clock access unless you add it with mount or os_access. I think that is exactly right: it is a sandbox for coordination code, not for running arbitrary programs.

Implementation
#

The whole change compared with the baseline is one capability (demos/monty.py):

from pydantic_ai_harness import CodeMode

code_mode = CodeMode(
    tools="all",
    max_tool_calls=50,
    resource_limits={"max_duration_seconds": 10, "max_memory": 128 * 1024 * 1024},
)

The defaults are 30 seconds and 256 MiB per snippet and 100 nested calls per run_code; I set tighter limits because this task needs very little.

You can read the code the model writes. run_code calls are promoted to RunCodeCallPart with a .code attribute, and the run_code result carries the nested tool calls in its metadata. The demo saves every snippet to output/monty/code/ and counts top-level and nested calls separately.

Result
#

You run it with make demo-monty. These are the four snippets gpt-6.1-sol wrote:

# snippet 1
brief = await get_tasting_brief()
brief

# snippet 2
catalogue = await search_whiskies()
catalogue

# snippet 3
import asyncio
ids = [w['id'] for w in catalogue]
results = await asyncio.gather(get_whisky_details(ids=ids), get_supplier_offers(supplier_id='S1', whisky_ids=ids), get_supplier_offers(supplier_id='S2', whisky_ids=ids), get_supplier_offers(supplier_id='S3', whisky_ids=ids))
results

# snippet 4 (shortened): checks the chosen four in code
chosen = ['W01','W02','W09','W10']
...
assert w['type'] == brief['required_type'] and w['abv'] <= brief['max_abv']
assert o['in_stock'] and o['stock_bottles'] >= bottles
Validation: VALID (checked against fixtures)
  constraints passed: 40/40

Metrics (monty, openai:gpt-6.1-sol)
  elapsed: 28.8s, model requests: 5
  tokens in/out: 22519/1600
  top-level tool calls: {'run_code': 4, 'final_result': 1}
  nested calls inside run_code: {'get_tasting_brief': 1, 'search_whiskies': 1, 'get_whisky_details': 1, 'get_supplier_offers': 3}
  run_code snippets: 4

All six lookups now go through Monty, and the plan is valid. But it took one request more than the baseline and two and a half times the input tokens.

Why Monty cost more here
#

The per-request numbers from the saved messages explain it:

RequestSnippetInput tokensReturned to the model
1get_tasting_brief()1,9601.2k chars
2search_whiskies()2,3092.9k chars
3details and three offers in parallel3,21713.3k chars
4check the chosen four7,2230.5k chars
5final plan7,810–

Three things stand out:

  • The model uses Monty like a REPL. It runs a small step, reads the result, and decides the next step. Variables survive between snippets (catalogue from step 2 is used in step 3), but the thinking still happens between snippets, in the model, one round trip each.
  • Every snippet ends with the raw value. brief, catalogue and results go back into the context unchanged, so nothing is saved on data.
  • Code Mode has a fixed cost per request. The first request costs 1,960 input tokens against 714 for plain tool calls: the run_code description carries the typed stubs of every tool and Monty’s rules. That overhead is paid on every request.

The chart shows this held for both models I tried, with one run each:

Bar chart of input tokens per run for scenario 1. gpt-6-luna: tool calls 8.8k, Monty 29.6k, Monty with a one-program prompt 18.7k. gpt-6.1-sol: tool calls 8.9k, Monty 22.5k, one-program prompt 13.1k.
Scenario 1 at a €120 budget, one run per variant and model. Plain tool calls were the cheapest every time; these are recorded runs, not a benchmark.

How I got it wrong first: asking for one program
#

If the problem is the step-by-step style, the obvious fix is to ask for a single program. The monty_program demo adds one instruction to demo 2: fetch, filter and summarise in one run_code call and return only a compact summary. Because the prompt differs, I treat it as an experiment, not as part of the comparison.

My first attempt with gpt-6-luna and the old €200 budget was the most expensive run of all, at 94k input tokens. Two things went wrong:

  • Monty type-checks code before it runs it. The model built dictionaries mixing strings, floats and nested dicts, so sum(x['line_total_cents'] for x in group) was rejected (No overload of function sum matches arguments), and so was the rewrite with +=. No tool ran, but every rejection cost a round trip.
  • “Compact summary” is up to the model. The program that did run returned every valid combination of four: 715 per supplier, 100,614 characters, which then stayed in the context for the remaining requests.

With gpt-6.1-sol and the €120 budget, it went better: three requests and 13.1k input tokens. The first program was again rejected, and this time the code was actually correct:

error[invalid-key]: Unknown key "whiskies" for TypedDict `SupplierOffersResult`
  |
7 | …d']: w for w in results[0]['whiskies']}
  |                  ---------- ^^^^^^^^^^ Unknown key "whiskies"
  |                  TypedDict `SupplierOffersResult` in union type `WhiskyDetailsResult | SupplierOffersResult`

asyncio.gather returns a list whose items the checker can only type as the union of all results, so results[0]['whiskies'] fails even though results[0] is the details. The model’s fix was to wrap the calls in async def ... -> Any helpers and annotate results: Any. The second program then did all lookups, the constraint filtering and the combinations in one go.

What I learned: one program does work mechanically, but a strict sandbox language has a price for loosely typed LLM code, and the size of what comes back is decided by the model, not by Code Mode. On a task this small, even the best run did not beat plain tool calls on tokens.

Scenario 2: where Monty does pay off
#

Scenario 1 is close to the worst case for Code Mode: six lookups that plain tool calling already batches, about 20 KB of data, and a decision that is mostly taste. I think Code Mode should help when the number of calls depends on the data, when there is far more raw data than the answer needs, and when exact arithmetic matters. So I built a second scenario around exactly that, using the real shape of the shop’s API.

Plan a tasting season for our whisky club of 30 members: four evenings with four whiskies each, 20 ml per member per whisky. Themes: Speyside, Islay, sherry casks, and world whisky (outside Scotland). Only single malts of at most 50% ABV, no whisky twice in the season. Buy each evening from one supplier with enough stock. Stay within the season budget, shipping included.

Real, messy data
#

The snapshot holds 504 whiskies between €20 and €80 from De Helm’s Store API, taken on 16 November 2026. Only facts are stored; ABV, “single malt” and “sherry cask” are parsed from the shop text and left null when unclear, and an unknown value never satisfies a rule. The data is messy in ways that need code: region is missing for 153 products, ABV for 76, bottle size is stated for only 31, and only 213 are confirmed single malts. The parser cost me one bug: the Dutch text “een van de eerste single malts” did not match \bsingle malt\b because of the plural.

Tools shaped like the real API
#

The tools mirror what the shop really offers: a paginated listing without ABV or bottle size, and batched details by ID.

def list_products(supplier_id: str, page: int = 1) -> ProductPage:
    """List one page of a supplier's whisky catalogue (PAGE_SIZE per page, pages start at 1).

    Listings have no ABV, bottle size or single-malt flag: use get_products for those.
    """

def get_products(supplier_id: str, ids: list[str]) -> ProductDetailsResult:
    """Full facts for up to MAX_DETAIL_IDS products at one supplier: ABV, bottle size, flags."""

With 50 products per page, there are 21 pages across three suppliers, about 187 KB of JSON. There are no filter parameters the real listing lacks, and both variants get the same instructions, without any hint to write one program. The scale has to make the case.

The budget is €540. A reference solver that only exists in the tests finds a valid season at €488.70; picking from the middle of each theme’s price range costs about €828. The tests check that a valid season exists, that the budget is close to the cheapest one, and that more than one valid season fits.

Result
#

You run both variants three times with make season-compare RUNS=3. Every run produced a valid season:

Tool callsMonty
Valid seasons3 / 33 / 3
Model requests (median)88
Input tokens (median)208k (187k–244k)125k (118k–133k)
Largest single request48.6k27.8k
Tool output sent back to the model150k chars70k chars
Listing pages read (of 21)14–1521, inside code
Elapsed (median)62 s65 s
Three small bar charts comparing tool calls and Monty in scenario 2. Input tokens per run: 208k versus 125k. Largest single request: 49k versus 28k. Tool output sent back: 150k versus 70k characters. Thin lines show the min–max range of three runs.
Median of three runs per variant with gpt-6.1-sol. Monty read the whole catalogue and still sent about half as much data back to the model.

This snippet, from the second Monty run, shows where the difference comes from:

import asyncio
pages = await asyncio.gather(*[list_products(supplier_id=s['id'], page=p) for s in brief['suppliers'] for p in range(1, (s['product_count']+49)//50+1)])
listings = {s['id']: [] for s in brief['suppliers']}
for page in pages:
    listings[page['supplier_id']].extend(page['products'])
{sid: [{'id':p['id'],'name':p['name'],'price':p['price_cents'],'stock':p['stock_bottles'],'region':p['region'],'country':p['country'],'labels':p['flavour_labels']} for p in sorted(ps,key=lambda x:x['price_cents']) if p['stock_bottles']>0 and p['price_cents']<=4500] for sid,ps in listings.items()}

One snippet requests all 21 pages in parallel, keeps them in a variable, and returns only the in-stock products up to €45, with seven of the fields. The full listings never enter the context. Its last snippet builds the four evenings and checks every rule with assert before the model writes its answer.

What I learned
#

  • Code Mode wins once data volume is the bottleneck. About 40% fewer input tokens and a request size almost halved, while reading the whole catalogue.
  • The saving does not come from fewer round trips. Both variants needed about eight requests, and Monty still worked in six or seven snippets. The difference is what each snippet returns.
  • Plain tool calling stopped reading early. It read 14–15 of the 21 pages, so it chose from part of the catalogue. Its plans were still valid, but it never saw some options.
  • Single runs are misleading. My first single run of each variant pointed the other way: Monty 176k against 129k for tool calls. With one run per variant, I would have drawn the wrong conclusion.

Monty and Smol: when the agent needs real files
#

Monty cannot import matplotlib or write a PDF, and it should not. For the tasting pack I gave the agent a second sandbox: a Smol microVM. The task gets one extra sentence: “Create a printable blind-tasting pack with cards labelled A–D, a separate host answer sheet, a shopping-list CSV and a flavour chart.”

It helps to be precise about what runs where:

flowchart TB
    subgraph HOST["Host process"]
        A["Agent loop and OpenAI calls"]
        D["Domain tools
(plain Python functions)"] subgraph MONTY["Monty sandbox"] S["run_code snippets
call tools, filter, compare"] end end subgraph VM["Smol microVM: no network, nothing mounted, no host env vars"] F["FileSystem and Shell tools
act on ctx.workspace"] P["make_pack.py
matplotlib, reportlab"] O["pack/: cards.pdf, answer_sheet.pdf,
shopping_list.csv, flavour_chart.png"] end A --> S S --> D A -->|"write_file, run_command"| F F --> P --> O O -->|"copied to the host before cleanup"| A

Adding Smol does not move the domain tools into the VM. Only the workspace tools, Shell and FileSystem, act on ctx.workspace, which is the VM.

Implementation
#

demos/monty_smol.py folds only the domain tools into Code Mode, so the file and shell tools stay ordinary tool calls:

Agent(
    config.get_agent_model(),
    instructions=INSTRUCTIONS + "\n" + PACK_INSTRUCTIONS,
    output_type=TastingPackResult,
    tools=list(DOMAIN_TOOLS),
    capabilities=[
        CodeMode(tools=DOMAIN_TOOL_NAMES, max_tool_calls=50,
                 resource_limits={"max_duration_seconds": 10, "max_memory": 128 * 1024 * 1024}),
        SmolSandbox(image=guest.GUEST_IMAGE, allow_internet_access=False),
        Shell(tools=["run_command"], allowed_commands=["python", "python3", "ls"],
              default_timeout=120, max_output_chars=20_000),
        FileSystem(),
    ],
)

I prefer Shell and FileSystem over the broader Coder capability here: writing and running one script needs nothing more. Note that the Shell allowlist is a guardrail, not the security boundary; the VM is.

A run never deletes its VM. Cleanup is your job, so the demo creates the VM backend itself, passes it to the run and deletes it in finally, also when the run fails:

backend = SmolSandboxBackend(image=guest.GUEST_IMAGE, cpus=guest.VM_CPUS,
                             memory_mb=guest.VM_MEMORY_MB, allow_internet_access=False)
try:
    await guest.install_guest_packages(backend)
    if not await guest.guest_network_closed(backend):
        raise RuntimeError("Guest VM has network access; refusing to continue.")
    result = await asyncio.wait_for(
        agent.run(f"{TASK}\n\n{PACK_TASK}", workspace=backend), timeout=RUN_TIMEOUT_SECONDS
    )
    copied = await export_workspace(backend, out_dir)
finally:
    if backend.ref is not None:
        await SmolSandbox().destroy(backend.ref)

backend.ref is set before the boot finishes, so even a cancelled boot is cleaned up. After my runs, smolvm’s local database listed zero VMs.

How I got it wrong first: preparing a VM without network
#

The report script needs matplotlib and reportlab inside the VM, and the VM has no network. Installing them in the host project does not help. My first plan was to boot a VM with network limited to PyPI, install the packages, stop it and pack it into a reusable .smolmachine image. The install worked; packing failed:

mkfs.ext4 not found. Install e2fsprogs or place a pre-formatted storage-template.ext4 in ~/.smolvm/

The alternative turned out better. make guest-image downloads Linux wheels on the host, 13 wheels and 42 MB, and every demo run copies them into a fresh VM and installs them with pip install --no-index. The guest never has network, not even during preparation. It is fast too: booting took 0.9 s with the image cached, copying the wheels with the smol SDK’s write_file 0.5 s, and the offline install took 3.5 s. A probe inside the VM confirms the network is closed: connecting to pypi.org fails with Temporary failure in name resolution.

Result
#

The agent planned in Monty, wrote make_pack.py with the file tool, ran it with the shell tool, and listed pack/ to confirm the four files. The host copies them out before the VM is deleted and checks them: valid PDF and PNG signatures, no whisky names on the cards, every name on the answer sheet, and CSV totals that match the plan.

Validation: VALID (checked against fixtures)
  ok   pack/cards.pdf: PDF; labels A-D present: True; whisky names leaked: none
  ok   pack/answer_sheet.pdf: PDF; whisky names missing: none (text search, may miss wrapped text)
  ok   pack/flavour_chart.png: PNG signature
  ok   pack/shopping_list.csv: 4 rows, line totals 11980 cents vs plan 11980
Bar chart generated by the agent inside the VM: four groups labelled A to D with sweetness, fruitiness, smokiness and intensity scores; smokiness rises from A to D. The title says illustrative estimates, 2026-11-15 shop snapshot.
The flavour chart the agent’s script produced inside the Smol VM, copied to the host before cleanup.
A printed blind-tasting card showing only a large letter A and lined space for notes.
A blind card from cards.pdf: no name, distillery or price.

The cards are what your guests see. The host gets the answer sheet, and it is my favourite file in the pack. It reveals the four whiskies in tasting order, explains the order (honey and vanilla, then sherry, gentle peat, and a smoky finish), and redoes the arithmetic: €26.95 + €27.95 + €34.95 + €29.95 = €119.80, leaving €0.20 of the €120 budget. It also reminds you to offer water and not to drive afterwards. The agent added that on its own; nothing in the prompt asked for it.

The host answer sheet: whiskies A to D revealed as Tamnavulin Double Cask, Glen Moray Sherry Cask, Bowmore 9 Years Old and Finlaggan Original, all 40% ABV, each with a one-line tasting note and flavour scores, followed by a shopping and volume check that adds the four prices to €119.80 against a €120.00 budget.
The host answer sheet from answer_sheet.pdf, generated inside the VM. Prices are from the 2026-11-15 shop snapshot; the flavour scores are illustrative estimates, not measurements.

With gpt-6.1-sol this took 8 model requests and 67k input tokens, plus about 5 seconds for the VM and the offline install. Writing a 7.8 KB report script costs far more tokens than the planning, so this demo is not comparable with the others, and it is not meant to be.

Things to be aware of
#

  • My first host-side PDF check passed for the wrong reason. ReportLab encodes page content as ASCII85 plus Flate and writes ~>endstream on the same line, so my naive “decompress and search” found no text at all, and “no whisky names on the cards” was trivially true. Test your validators with files that should fail.
  • smol.Machine.list() only works against Smol Cloud. For local VMs, store the ref yourself.
  • File operations through the generic workspace API move 64 KiB per command. For bulk copies, use the SDK handle from backend.get_sandbox().

Measuring honestly
#

A comparison like this is easy to bend, so a few rules kept me honest:

  • Validate independently. The agent’s own explanation is not evidence; the validator recomputes every plan from the data.
  • Separate top-level from nested calls. Code Mode reports nested calls in the run_code metadata, so you can see that the lookups did not disappear; they moved.
  • Never estimate. A value the framework does not report is stored as null, not derived from text length.
  • Repeat runs. One run per variant pointed the wrong way in scenario 2.
  • Keep the comparison fair. Same tools, same data, same instructions; prompt changes are labelled as experiments.

What I take from these experiments
#

  1. Code Mode pays off when code can shrink data. On a 504-whisky catalogue, Monty used about 40% fewer input tokens and halved the largest request, while reading more of the data.
  2. On small tasks, it costs more. Each request carries roughly 1.2k extra tokens of tool description, and plain tool calling already batches independent lookups. Measure your own workload before switching.
  3. The model decides how well Code Mode works. It tends to use Monty like a REPL and to return raw values. The saving comes from snippets that filter before they return, not from fewer round trips.
  4. A strict sandbox has a cost. Monty’s type checker rejects loosely typed code before any tool runs; that is good for safety and costs a round trip per rejection.
  5. Use the right sandbox for the job. Monty for coordinating tool calls in the host process, a Smol VM for files and full programs with real libraries, with no network and explicit cleanup.
  6. Make the scenario prove something. A budget that never binds, a single run, or a validator that cannot fail all produce convincing numbers that mean nothing.

References
#

Related