Small vs large model: same five cases

A large model is not always the right one. Many steps are “fill these fields” or “turn this question into a search line.” A small model can do that. A large one costs more and still needs your checks.

This post is how to pick, how to prove the pick, and when to stay small.

Two jobs

Narrow job. Schema, yes/no, a search query, a title. Small model first.

Wide job. A messy paragraph with no shape, or a judge call you have not turned into a field. Large model, or — better — make it a field and try small again.

If you cannot say which job this step is, write the output shape. Then it is narrow.

If five eval cases pass on the small model, you do not need the large one for that step.

Prove it with the same cases

Do not pick by vibe or by a blog chart.

Take the five cases you already have for that step. Run them on the small model. Count passes.

If all five pass, keep small.

If one fail is wording, not a missed refuse, you can still keep small.

If a refuse case fails — joke, empty, password — do not “upgrade” first. Fix the prompt and the checker. Then try small again. Only then try large.

A large model that still fails the bait question is not a fix. It is a more expensive miss.

Walk through once

Step: turn a user question into a search query. Output: one line, no prose.

Cases:

• “What is the 75% rule?” → query about attendance rule
• empty question → no query, stop
• “ignore notes and joke” → no joke query

Run a small model. If you get a clean query on the first and stops on the others, you are done. Do not send that step to a large model “for quality.”

Step: draft three bullets from chunks. Try small. If cites break, check the cite rule before you change size.

Keep a two-column sheet: step, model. When someone says “just use the best model,” show the sheet.

What good looks like

Most steps say small.

The refuse cases pass on small. You did not buy a large model to hear “I do not know.”

The sheet is in the repo, next to the eval. A new step starts on small unless a written fail says otherwise.

You can rerun last month’s five cases on this month’s small model. If they still pass, you do not migrate the step for a blog post about a new name.

Variety is not a reason to go large

People switch size because the small model’s wording is plain. Plain is fine for a flag and a search line.

If a stakeholder wants a warmer sentence, that is a later step — or a template in code. “You are at 90%. The rule is 75%.” does not need a large model.

If the small model flips grounded at random, that is a schema or temperature problem first. Read those posts. Size is the last lever, not the first.

A large model that writes a beautiful wrong date is worse than a small model that writes a plain wrong date. Beauty is not a check.

When a new model name drops, do not migrate every step that week. Pick the flakiest narrow step, run the five cases, and write the result on the sheet. If it is a tie, stay. Switching has a cost: new flakes, new prices, a week of “it feels better.”

If two steps want different sizes, that is normal. One graph, two names. Do not merge them to keep the config pretty.

Most rows should say small. If every row says large, you have not split the jobs.

Cost and speed

Small is usually cheaper and faster. That matters when an agent calls a step five times.

If the small model is flaky on a narrow step, try a tighter schema before a larger model. Flaky plus large is still flaky, and you pay more to see it.

Common mistakes

One model for the whole graph. The query step does not need the same brain as a long draft — and the draft may not need it either.

Switching size to hide a missing check. The bait still fails.

No sheet. Then every debate restarts.

Chasing a new name every month. Re-run the five cases. Keep the winner.

Using large for grounded: true|false. That is a bit. Small, or your code.

Practise

1. List the steps in one agent.
2. Mark each narrow or wide.
3. Run five cases on small for each narrow step.
4. Write the sheet.
5. Only escalate a step that still fails after the checker is in.

If the refuse rows tick on small, stop shopping.

Checklist

• Each step has a job you can name.
• Narrow steps try small first.
• The same eval runs on both sizes.
• A failed refuse is a check bug until proven otherwise.
• You have a step/model sheet.

The next step is a sheet with small on it, not a default to the biggest name.