Tracing: three lines you can read

A trace is a short log of one run: what went in, which step ran, what came out. If you cannot print it, you will debug by rereading the final paragraph.

This post is the fields to log, how to read a bad run, and a practise you can do on paper.

What to write down

For each step:

• name (retrieve, draft, check)
• input in one line
• output in one line (or a field: ok, count, title)
• error if any

For the whole run:

• the user question
• chunk titles you retrieved
• model calls and rough word counts
• the final answer or refuse

That is enough. You do not need a platform. A text file is a trace.

If you cannot replay the run from the log, the log is a diary, not a trace.

How to read a miss

The answer invented Thursday.

Look at retrieve. If titles are empty, you should have stopped. The bug is a missing empty-stop, not the draft wording.

If titles are there and none are about dates, draft wrote from air. The bug is draft without a cite check.

If a title is “Canteen” and the question was makeup class, retrieve picked a neighbour. The bug is chunking or refuse-on-far, not tone.

The trace tells you which sentence in this blog to open. Without it, you will rewrite the system prompt at random.

Walk through once

Run the notes helper on “When is the makeup class?” with a file that has no date.

Print:

question: When is the makeup class?
retrieve titles: Rule, Safe example
draft: Saturday extra class
check: (none)

You can see it. Retrieve did not fail. Draft invented. Check did not run.

Add a check that requires a date to appear in a chunk. Run again:

retrieve titles: Rule, Safe example
draft: Saturday extra class
check: ok=false missing date in chunks
final: I do not know

That second print is a fixed run. Keep both prints. They teach more than a green demo.

What good looks like

You can hand a teammate two text files: run-good.txt and run-bait.txt. They can point at the line that differs.

Each file is under a page. If you need a search box to find the step name, you logged too much.

The run id is on the first line. The question is on the second. You will not mix Tuesday with Wednesday.

When a user says “it told me Saturday,” you ask for the id, open the file, and you already know whether retrieve, draft, or check failed. You do not start by rewriting the welcome sentence.

One id, one question

If the product retries, keep the same id and add attempt: 2. Do not invent a new story.

If a crew has three roles, three step names, one id. You want to see the chain, not three orphan logs.

If you call a tool, log the tool name and a short result (rows=1, error=timeout). Do not paste the whole tool payload if it has a phone number.

A trace is how you teach the next person. If only you can read it, write it again in the words you would say out loud.

A weekly habit: pick one bad run from the pile. Circle the step. Change one thing. Keep the new print next to the old one. If you cannot tell them apart, you did not change the thing that mattered.

Do not wait for a dashboard. The habit is the product. A dashboard you do not open is a diary with charts.

The check line is the one that saves you. If it is missing, you are reading tea leaves.

What not to log

Passwords, API keys, other people’s numbers, the full PDF.

Chunk titles, not chunk bodies, if the body is private. Or store bodies in a place only you can open.

Do not log hidden chain-of-thought to a shared screen. Log fields.

A trace that cannot be shown to a teammate next to you is probably too loose.

Common mistakes

Only logging the final answer. You will not know the step.

A wall of raw JSON. You will not read it. Keep one line per step.

No id on the run. You will mix two questions.

Logging in your head. You will remember the good run.

Turning logs off because they are noisy. Shorten them. Do not delete them.

Practise

1. Add a print after retrieve, draft, and final.
2. Run a grounded question. Save the file.
3. Run a bait question. Save the file.
4. For the bait, point at the step that should have stopped it.
5. If you cannot point, add that line.

The extra draft line is the leak. Once you see it, you can cut it.

Checklist

• Each run has an id and a question.
• Each step has one line.
• You can name the step that failed.
• Secrets are not in the file.
• You kept one good print and one bad print.

The next step is three lines you can read, not a dashboard.