Excelrs
o1: the model that thinks before it answers
Launched 12 September 2024. AIME and GPQA numbers are theirs. Thinking is a second bill — route only the hard case.
Sandeep Jaiswal·12 September 2024 · 5 min read
On 12 September 2024 OpenAI launched o1-preview and o1-mini. The specialty is not faster chat. It is a model that spends time thinking before it answers.
This post is what that means, which exam scores they published, and when you should pay for thinking — and when you should not.
What launched
GPT-4o answers in one go. o1 is trained with reinforcement learning to write a long internal chain of thought, then a final answer. They said performance rose when they spent more compute in training, and again when they let it think longer at test time.
ChatGPT Plus and Team could pick o1-preview and o1-mini that day. Caps were tight: 30 messages a week for preview at launch. The full o1 arrived later, on 5 December 2024. The idea did not change.
They hid most of the chain from the user. You see that it thought, not every scratch line. That is a product choice, not a bug in your client.
Thinking is a second bill. Use it when the first guess is the thing that fails.
What the specialty is
Hard problems with a right answer: contest math, a proof, a nasty bug, a science item a guess will miss.
Not: “rewrite this email.” Not: “three bullets from a file.” Those want a fast model plus your checks.
Human raters in their test preferred o1-preview on math, code, and analysis. They did not prefer it on some ordinary language tasks. That is the fork.
Numbers they published
On the 2024 AIME, a US high-school olympiad qualifier, they said GPT-4o averaged about 12% (1.8 of 15). o1 averaged about 74% (11.1 of 15) with one try, 83% with a vote of 64 tries. A re-rank of many samples reached about 93%, which they said sat with the top 500 students and above the USAMO cutoff.
On GPQA Diamond (chemistry, physics, biology items written to stump people), they said o1 beat the PhD volunteers they hired on that set. That is not “smarter than a PhD at a job.” It is better at those items.
On Codeforces-style contests, they said GPT-4o sat around Elo 808 (about the 11th percentile). A programming-tuned o1 variant sat around 1807 (about the 93rd). o1 itself they put in the 89th percentile on competitive programming questions.
| Exam | GPT-4o (their figure) | o1 (their figure) |
|---|---|---|
| AIME 2024, one try | ~12% | ~74% |
| GPQA Diamond | lower | first to beat their PhD panel on that set |
| Codeforces-style | ~11th percentile | ~89th (o1 family) |
Run your own cases. Contest scores are not a notes helper.
Walk through once
Question: “I have 36 of 40. The rule is 75%. How many more can I miss?”
o1 may write a careful paragraph. A calculator already knows: 40 × 0.75 = 30, so 6 more absences stay legal. Do not spend thinking tokens on that.
Now a real o1 job: a 12-line function that drops the last attendance row on leap years. You have a failing test. Paste the function and the test. Ask for the smallest change. Then run the test yourself. Thinking helps. Shipping without the test does not.
A bait: “When is the makeup class?” and no date in the file. A reasoning model can invent a tidy Thursday. Thinking does not add a source. Retrieve, or refuse.
What good looks like
The picker is not stuck on o1.
A hard case has a time budget and a token cap in the trace.
The AIME figure is written as “theirs, 12 September 2024,” not as your product KPI.
A refuse still happens after a long think. Fluent scratch is not a source.
The hourglass is the product. If the card is a percentage, skip the glass.
Cost and delay
More thinking is more output tokens and more wait. o1-mini exists because not every puzzle needs the large one.
If five eval cases already pass on a small fast model, stay there. If one case is a contest-style miss and the others are fine, route only that case.
A loop that “thinks again” is two tickets. Cap it.
Common mistakes
Defaulting every chat to o1. You will wait, pay, and get a heavier email.
Treating AIME as your product score. Your product is five cases you wrote.
Showing hidden chain-of-thought on a shared screen. They hid it for a reason. Log fields, not scratch.
Assuming thinking stops hallucination. It can make a wrong story more fluent.
No time budget. A student tool that thinks for a minute on “hi” is broken.
Practise
- One arithmetic question. Time a fast model. That is the baseline.
- One failing unit test. Time o1 or a current reasoning model. Run the test after.
- One file question with no answer in the file. Confirm refuse.
- Write a router: think only if the case is marked hard.
- Put the wait and the token count in the trace.
If every arrow goes to think, you do not have a router.
Checklist
- You can say o1 in one line: it thinks before it answers.
- You know AIME and GPQA numbers are theirs, from launch.
- Easy counts stay on a fast model.
- A refuse still needs a source check.
- Thinking has a cap.
The next step is one failing test, not a new default in the picker.