Excelrs
GPT-4o: one model for text, picture, and voice
Launched 13 May 2024. Omni means one network, not three taped together. When a photo helps, and when a calculator still decides.
Sandeep Jaiswal·13 May 2024 · 5 min read
On 13 May 2024 OpenAI launched GPT-4o. The “o” is for omni: one model that takes text, pictures, and sound, and can answer in those forms. Before this, voice in ChatGPT was three models taped together.
This post is what that change is, which numbers they published, and when omni helps a student tool — and when it does not.
What launched
The old voice path was: speech to text, then a text model, then text to speech. The middle model never heard the laugh or the second speaker. It only saw words.
GPT-4o is one network for those inputs. OpenAI said it can answer audio in about 320 milliseconds on average, down to 232 milliseconds. That is conversation speed, not a voicemail.
On English text and code they said it matches GPT-4 Turbo. They said it is better on many other languages, and about twice as fast and half the price of GPT-4 Turbo in the API. A new tokenizer used fewer tokens on Indian languages — Gujarati about 4.4 times fewer in their example.
At launch, ChatGPT got text and pictures. Free users got the model with a cap. The API got text and vision the same day. Native audio out came later, and only with set voices.
One model that sees the picture is not the same as three models that only pass text.
What the specialty is
The specialty is not “smarter essays.” It is one brain on a photo plus a question, or a short spoken turn, without a separate transcriber eating the tone.
A menu photo in another language. A whiteboard. A screenshot of an error. Those are the jobs this launch was for.
A long proof with no picture still belongs to a reasoning model. Omni does not replace that.
Numbers they published
| Claim (OpenAI, 13 May 2024) | Meaning for you |
|---|---|
| Matches GPT-4 Turbo on English text and code | Do not expect a leap on a notes essay |
| Better vision and audio than prior GPT-4 | The picture is the new input |
| ~320 ms average on audio | Voice can feel live |
| 2× faster, ~50% cheaper than GPT-4 Turbo in the API | Same text job may cost less |
| Gujarati 4.4× fewer tokens in their demo | Hindi and neighbours got cheaper too |
These are their measurements. If you need a number for your product, run your five cases.
Walk through once
You have a photo of a handwritten attendance sheet: 36 ticks out of 40.
Wrong use: send only “how many can I miss?” and hope.
Right use: send the photo plus “Read present and total. Do not invent a date.” Then your code still does 36/40 >= 0.75. The model reads the sheet. The if decides.
Now try a voice note: “I was in 36 of 40, can I sit?” Same rule. If you only transcribe first, you lose a mumbled number. Omni can hear it. You still check the count in code.
A bait: a photo of a canteen menu. Question: “What is the 75% rule?” The photo has no rule. The model must not invent one from the samosas.
One chip. If you still run three models, you are on the old voice path.
When not to reach for it
If the input is already a clean number in a table, do not send a photo of the table. Read the row.
If you need a cite from a handbook, omni does not replace retrieve-and-open. A pretty reading of a screenshot is still a reading. Open the file.
If the job is a hard contest problem, this launch is the wrong specialty. That is the o1 post.
Audio that can impersonate a person is a product risk. At launch they limited voices. Treat a cloned voice as out of scope unless you have a human in the loop.
Common mistakes
Calling it a new brain for every task. The text leap was small. The input leap was the point.
Trusting a photo read without a check. “36” misread as “30” flips the exam seat.
Piping voice through a separate transcriber “to be safe.” You threw away the specialty.
Skipping the language token win. A Hindi notes helper got cheaper. Measure that.
Shipping live video on day one. They did not. Neither should a student app.
Practise
- One photo of a real sheet. Ask for present and total only.
- Compare with a typed version of the same sheet. The counts should match.
- A photo that has no attendance. Confirm refuse.
- Time a short voice question if you have the voice path. Note the lag.
- Keep the
iffor 75% in code.
The photo is input. The seat decision is still arithmetic.
Checklist
- You can say omni in one line: one model for text, picture, sound.
- You know the old voice path was three models.
- Official numbers are labelled as theirs.
- A photo job still has a check.
- Hard proofs are not why you picked this model.
The next step is one photo plus two fields, not a new personality.