How we made Speech-to-Text noticeably better with 90 data points and two prompts
We use Speech-to-Text every day. Instructions to the AI are spoken — not typed. It is faster, more natural, and you can walk around while doing it. Anyone working seriously with AI and writing a lot of prompts quickly notices: your hands are usually the bottleneck. Speech-to-Text fixes that.
The problem: Whisper, the open-source model we use for transcription, initially had no idea what “DEVI Runner”, “boostN” or “MindVaults” were supposed to be. The AI heard “Debbie Runner”, “Boosten” and “Mindwalls”. And those errors landed straight in the next prompt — and therefore in the result.
This was not a theoretical problem. It was a daily annoyance.
The real problem: STT errors propagate silently
When Whisper transcribes “Heiko” instead of “Haiku”, you often do not notice immediately. The follow-up prompt still sounds plausible. The AI interprets it somehow. But the result is worse than it should be — and you do not know why.
The tricky thing about Speech-to-Text errors: they are invisible. You speak, you get an answer, and you never think that something went wrong between your mouth and the AI input. Especially with technical terms, proper names and domain-specific vocabulary, the small Whisper model is structurally overwhelmed — not because it is bad, but because it knows nothing about your context.
Our hypothesis: if we know the most frequent errors and feed the right keywords to the model as a hint upfront, we should be able to fix this systematically — no training, no fine-tuning, no infrastructure.
The approach: measure before you optimise
Whisper has a parameter called initial_prompt. You can pass the model a keyword list before transcription — essentially context telling it: “You are about to hear about these things. Know them.” This is not fine-tuning and not real model training — it is a deterministic hint that works immediately and needs no training pipeline.
But which keywords belong in there? Guessing would be waste.
So we measured — with AI support.
Step 1: have every STT session rated
We built a rating prompt that produces a structured analysis for every transcription:
You are evaluating an automated speech transcription (Speech-to-Text).
I will give you:
- the original text (what the user meant / the corrected text)
- the STT transcription (Whisper output, uncorrected)
Rate the transcription on a 1–10 scale in these categories:
- word_accuracy
- english_terms
- german_technical_terms
- completeness
Also note all concrete errors in the field error_notes:
- Format: "Wrong instead of Right" — e.g. "Davy Runner instead of DEVI Runner"
- Mark word-splitting errors with "(word split)"
- Mark sense-distorting errors with "(distorting)"
Return the result as a single CSV row.The outcome: a growing CSV file, row by row, session by session.
A look at the raw data
To give you a feel for what we actually measured — here are ten real CSV rows, unfiltered:
| Timestamp | Word acc. | EN terms | DE terms | Complete | Errors | |---|---|---|---|---|---| | 05.04. 01:25 | 7 | 4 | 8 | 9 | Cloth Session → Claude Session; son net → Sonnet; Heiko → Haiku | | 05.04. 01:30 | 6 | 5 | 7 | 8 | Clot → Claude; Boosten → boostN; Fußsch alter → Fußschalter (3x) | | 06.04. 00:55 | 7 | 8 | 6 | 9 | Prozeltiere → Prozeduren (distorting) | | 06.04. 18:25 | 6 | 5 | 6 | 9 | Herdbeet → Heartbeat; abstuerbt → abstirbt; schuetzt → schickt | | 06.04. 20:56 | 7 | 5 | 7 | 9 | Hammersboden → Hammerspoon; Drücker → Fußschalter | | 10.04. 10:00 | 6 | 4 | 7 | 8 | Speed to Text → Speech to Text; CL I → CLI; lock dich ein → logg dich ein | | 10.04. 10:20 | 6 | 4 | 6 | 9 | Whispernitzen → Whisper nutzen; Boosten Jason → boostn.json; Enft → .env | | 11.04. 00:15 | 7 | 5 | 8 | 9 | Debbie Runner → DEVI Runner; Sosommengesetzte → zusammengesetzte | | 12.04. 00:10 | 6 | 7 | 6 | 9 | Transpiration → Transkription; Beißbetext → Beispieltext | | 12.04. 16:15 | 7 | 8 | 6 | 8 | Transklippierung → Transkribierung |
A few are real classics: “Prozeltiere” instead of “Prozeduren” (procedures) — Whisper guessed phonetically and went completely off-piste. Or “Hammersboden” instead of “Hammerspoon” (a tool name) — the model simply lacked context. And “Whispernitzen” instead of “Whisper nutzen” (“using Whisper”) shows how fluent fast speech ends up as one made-up word.
Completeness scores are interesting too: almost every session lands at 8–9 even when individual words go wrong. The overall meaning usually still survives — but with noise that accumulates downstream.
Step 2: evaluate after ~90 entries
After roughly 90 annotated sessions — collected over about a week — we fed the CSV into a second prompt:
Analyse the errors and answer:
1. Top error patterns — which words are most frequently mis-transcribed?
2. Glossary recommendation — produce a list for Whisper initial_prompt (max. 15 terms).
3. Quality trend — does quality improve or worsen across sessions?
4. Unsolvable errors — what cannot be fixed by a glossary?What we learned
The errors fell into clear clusters straight away:
| Error class | Examples | |---|---| | Proper names & brands | Claude → Kloat / Clot, DEVI Runner → Davy / Debbie Runner, boostN → Boosten | | Model names | Sonnet → son net, Haiku → Heiko | | Tech terms | JWT → JVT, Heartbeat → Herdbeet, CLI → CLi | | Word splitting | Fußsch alter, ab geschickt, auf rufen | | Hallucinations | Transpiration instead of Transkription, Prozeltiere instead of Prozeduren |
Proper names and model names: solvable through a glossary. Word splits and hallucinations: need a different approach (correction layer, larger model).
The glossary that came out of the analysis:
DEVI Runner, boostN, MindVaults, Sonnet, Haiku, Heartbeat, Fußschalter, Transkription, Claude, CLI, JWT, Commit, Branch, Node, WorktreeThe surprising insight: keyword count matters
This is something we did not expect — and it is relevant to anyone using Whisper with initial_prompt.
For Whisper-small — the small, fast model — there is a clear sweet spot: under 15 keywords in the glossary. As soon as you push more in, general transcription quality noticeably degrades. The model loses focus, errors accumulate, and the gain from the keyword list is eaten up by new errors.
This is an important hint for anyone tuning Whisper-small with their own terminology: less is more. Do not try to solve every error at once — prioritise.
With Whisper-medium (the mid-size model) the capacity window is significantly larger. There you could comfortably extend the keyword list — something we are keeping in mind as a next step. Our gut feeling: with the mid-size model, baseline quality is already better, and the glossary could carry 20–25 terms without degrading general recognition.
The result
After deploying the glossary: proper names get transcribed reliably and correctly. “DEVI Runner” stays “DEVI Runner”. “Claude” stays “Claude”. “Haiku” stays “Haiku”.
No training. No fine-tuning. No infrastructure. Two AI prompts, one CSV, 90 data points — and a noticeably better Speech-to-Text system, tailored to our own context.
The approach scales: anyone working with AI regularly who has their own technical terms, product names or tooling can copy this process one-to-one. The result is a personalised keyword glossary — not generic fine-tuning, but precisely matched to your own workflow.
You can do this yourself
The nice thing about this approach: it is not tied to our infrastructure. All you need is:
- An STT system with Whisper (or a compatible model)
- The rating prompt — one CSV row per session, scored by any AI
- The evaluation prompt — after ~50–100 sessions, the automated keyword glossary follows
If you host Whisper yourself or use it via an API, you can fill the initial_prompt field directly with your keyword list. No additional infrastructure, no fine-tuning needed.
Or the faster route: if you use boostN, the CLI ships Speech-to-Text directly — including glossary integration. The account is free, setup takes a few minutes. Just try it, build your own glossary, and noticeably improve your voice-driven workflow.
The two prompts from this project are at the end as a copy-paste template.
Prompt 1: STT rating (per session)
You are evaluating an automated speech transcription (Speech-to-Text).
I will give you:
- the original text (what the user meant / the corrected text)
- the STT transcription (Whisper output, uncorrected)
Rate the transcription on a 1–10 scale in these categories:
word_accuracy — how many words were transcribed correctly?
english_terms — how well were English technical terms recognised?
german_technical_terms — how well were German technical terms and proper names recognised?
completeness — how completely was meaning preserved?
Note all concrete errors in the field error_notes:
- Format: "Wrong instead of Right"
- Word-splitting errors: "(word split)"
- Sense-distorting: "(distorting)"
- No errors: "No errors"
Return the result as a single CSV row:
timestamp,word_accuracy,english_terms,german_technical_terms,completeness,word_count,error_notes
Original text:
[ORIGINAL HERE]
STT transcription:
[WHISPER OUTPUT HERE]Prompt 2: Evaluation & glossary recommendation (after ~90 sessions)
I have a CSV of annotated speech transcriptions.
Columns: timestamp, word_accuracy, english_terms, german_technical_terms,
completeness, word_count, error_notes
Here is the CSV:
[CSV HERE]
Analyse the errors and answer:
1. Top error patterns — which words are most frequently mis-transcribed?
Group by: Proper names/brands · English technical terms · Word splits · Hallucinations
2. Glossary recommendation — produce a comma-separated list for Whisper initial_prompt.
Max. 15 terms. Prioritise by error frequency.
3. Quality trend — does quality improve or worsen across sessions?
4. Unsolvable errors — what cannot be fixed by a glossary?Entdecke mehr
Working with AI: one word, two actions — and an hour of debugging
One word, two meanings: I blocked a Git commit, the AI heard deploy. Why a shared vocabulary decides everything when you work with AI.
GlossarSpeech-to-Text (STT)
Speech-to-Text refers to the automatic conversion of spoken language into text by an AI model. In an AI workflow, STT replaces the keyboard as the input channel — what matters is model size and domain vocabulary.
LexikonAutomating AI Workflows
No-/low-code automation with Make, Zapier and n8n: building blocks, use cases for agencies and SMBs, pricing models and the costliest pitfalls.