We Benchmarked Our Speech Recognition Against Whisper in 59 Languages
Most transcription tools tell you they support 100+ languages. Almost none tell you how well.
That is a convenient gap, because supporting a language and being any good at it are different things — and the difference is largest exactly where you can least afford it, in languages with little training data behind them.
So we measured. Here is the whole thing: every number, the method, and the languages where we lose.
The headline
Across 57 languages scored on both sides:
Whisper wins English. We win everything else that both systems can transcribe.
And there are two languages Whisper cannot attempt at all — it rejects Kyrgyz and Kinyarwanda as unsupported. We transcribe Kyrgyz at 5.8% and Kinyarwanda at 22.1%.
How this was measured
The data. FLEURS — Google's multilingual speech benchmark. It is the same sentences read aloud in every language, which is the property that makes a cross-language comparison meaningful at all: a language is not "harder" here just because its test set is harder. Twenty clips per language, sixty where we needed a larger sample. The baseline. OpenAI's `whisper-large-v3`, the model behind a large share of the captioning market, run on the identical audio files. The metric. Word error rate — insertions plus deletions plus substitutions, over the number of words in the reference. Lower is better. 5% means one word in twenty is wrong. The normalisation, which matters more than it sounds. Both transcripts are lowercased, stripped of punctuation, and have numbers put into one consistent form before scoring.That last step is not cosmetic. We write "4 years"; the FLEURS reference says "four years", and so does Whisper. Without normalising, we get charged a full word error every time we format a number the way a subtitle should be formatted — on 14% of all clips. That single artefact was worth about a point of WER overall, and far more in some languages: Finnish 6.2% → 2.8%, Catalan 4.5% → 1.3%.
We normalise both sides. Scoring it the other way would have flattered Whisper for no reason other than that it spells numbers out.
Full results
Word error rate, lower is better. Both sides normalised identically.
| Language | Harmar | Whisper large-v3 | Clips |
|---|---|---|---|
| Afrikaans | 9.1% | 35.6% | 20 |
| Albanian | 8.8% | 56.0% | 60 |
| Armenian | 9.3% | 38.7% | 20 |
| Bashkir | 19.9% | 99.7% | 60 |
| Basque | 6.2% | 40.3% | 60 |
| Belarusian | 3.0% | 41.1% | 20 |
| Bosnian | 6.4% | 16.0% | 20 |
| Bulgarian | 2.3% | 9.6% | 20 |
| Catalan | 4.1% | 5.5% | 20 |
| Croatian | 1.9% | 10.1% | 20 |
| Czech | 0.8% | 12.8% | 20 |
| Danish | 6.2% | 12.7% | 20 |
| Dutch | 3.1% | 5.8% | 20 |
| English | 3.3% | 2.5% | 20 |
| Estonian | 4.6% | 17.2% | 20 |
| Finnish | 3.6% | 8.8% | 20 |
| French | 1.8% | 3.4% | 20 |
| Galician | 2.5% | 13.8% | 20 |
| Georgian | 6.7% | 66.0% | 20 |
| German | 2.9% | 3.1% | 20 |
| Greek | 2.4% | 5.8% | 20 |
| Hausa | 12.9% | 86.0% | 20 |
| Hungarian | 3.9% | 14.6% | 20 |
| Icelandic | 3.6% | 37.6% | 20 |
| Indonesian | 3.0% | 6.7% | 20 |
| Italian | 0.7% | 1.5% | 20 |
| Kazakh | 4.0% | 39.3% | 20 |
| Kinyarwanda | 22.1% | not supported | 55 |
| Kyrgyz | 5.8% | not supported | 20 |
| Latvian | 3.6% | 17.9% | 20 |
| Lingala | 18.7% | 64.0% | 20 |
| Lithuanian | 4.5% | 26.7% | 20 |
| Luxembourgish | 23.8% | 86.3% | 20 |
| Macedonian | 2.9% | 12.8% | 20 |
| Malay | 2.9% | 8.9% | 20 |
| Maltese | 5.4% | 68.7% | 20 |
| Mongolian | 12.3% | 88.9% | 20 |
| Norwegian | 4.1% | 8.1% | 20 |
| Polish | 1.6% | 4.3% | 20 |
| Portuguese | 2.0% | 3.0% | 20 |
| Romanian | 1.9% | 8.1% | 20 |
| Russian | 1.6% | 3.4% | 20 |
| Serbian | 2.5% | 91.6% | 20 |
| Shona | 15.6% | 113.0% | 20 |
| Slovak | 0.5% | 6.4% | 20 |
| Slovenian | 6.0% | 19.6% | 20 |
| Somali | 40.5% | 95.1% | 20 |
| Spanish | 3.2% | 4.3% | 20 |
| Swahili | 5.3% | 30.7% | 20 |
| Swedish | 5.5% | 9.3% | 20 |
| Tagalog | 6.1% | 11.4% | 20 |
| Tajik | 5.9% | 95.7% | 20 |
| Tatar | 9.4% | 92.7% | 60 |
| Turkish | 3.8% | 4.1% | 20 |
| Turkmen | 44.6% | 99.3% | 60 |
| Ukrainian | 4.3% | 10.5% | 20 |
| Uzbek | 7.6% | 85.0% | 20 |
| Vietnamese | 3.1% | 6.9% | 20 |
| Yoruba | 75.2% | 100.2% | 20 |
Where the gap is largest
The pattern is not subtle: the fewer resources a language has, the wider the gap.
| Language | Harmar | Whisper |
|---|---|---|
| Serbian | 2.5% | 91.6% |
| Uzbek | 7.6% | 85.0% |
| Tatar | 11.1% | 90.6% |
| Mongolian | 11.7% | 87.1% |
| Hausa | 12.9% | 86.0% |
| Maltese | 7.4% | 69.5% |
| Georgian | 6.7% | 65.1% |
| Albanian | 8.8% | 56.0% |
At 85–92% word error rate, a transcript is not a rough draft. It is noise — there is nothing in it worth correcting, and you would be faster typing the whole thing yourself.
Meanwhile in French, German, Spanish and Italian, both systems are in low single digits and the practical difference is small. If you make videos in a major European language, most tools will serve you. The reason to care about this table is if you do not.
Where we lose, and where we are simply bad
A benchmark that only reports its wins is marketing. Ours:
English — Whisper wins. 2.5% against our 3.3%. English is the most over-served language in speech recognition and we are not going to beat a model trained overwhelmingly on it. German, Italian, Turkish — effectively a tie. We are ahead by a few tenths of a point, which is not a difference anyone experiences. Yoruba — we are bad. 75.2% as transcribed. Strip the diacritics before scoring and it falls to 19.7%, which tells you exactly where the failure is: we hear the words and we get the tone marks wrong. Yorùbá tone marks change meaning, so this is a real defect and not a formatting quibble. Whisper is worse still (100.2%), and that is no defence. We are fixing it, and until we have, we do not offer a Yoruba page. Turkmen (44.6%) and Somali (40.5%) are poor in absolute terms, even though Whisper is at 99.3% and 95.1%. Being twice as good as unusable is not the same as being good.The code-switched result, which matters more
Everything above is one language at a time. That is not how a great many people actually talk on camera — Spanglish, Taglish, Hinglish, Armenian with Russian and English — and mixed speech is the case this product was built for.
So we measured that too, on CS-FLEURS, using the subset of real recorded bilingual speech rather than the synthesised portion. That distinction matters and most published numbers skip it: text-to-speech switches languages cleanly, and a person switching mid-breath does not. A synthetic score is a ceiling, not an estimate.
Hindi–English:
That gap has a documented cause. Published benchmarking finds Whisper responds to a language switch by translating rather than transcribing, which sidesteps the question of which language the sentence is in and produces a caption of something nobody said. Researchers also record word-level language identification failures, where a word survives but is mapped phonetically into the wrong language — the error that reads as a typo and is hardest to catch.
The part we did not expect
| Word error rate | |
|---|---|
| Hindi, spoken on its own | 12.4% |
| Hindi mixed with English | 7.9% |
We are not claiming a general law from one language pair. But it is the behaviour you would expect from a pipeline built around code-switching from the start rather than adapted to it afterwards, and it is the opposite of what a system that picks one language and commits to it does.
What this does not measure
Three limits, and the third is the one we care most about.
FLEURS is read speech. Someone reading a written sentence in a quiet room. Your videos are spontaneous, in a kitchen, with music. Every system scores better here than on real footage, ours included. Use these numbers to compare systems, not to predict what you will get. It is one voice at a time. No overlapping speakers, no crosstalk. It is monolingual. Every clip is one language, start to finish — which is why the code-switched result above exists as a separate measurement, and why it is the more important of the two for anyone who mixes languages on camera. And the code-switched table covers one pair so far. Hindi–English is measured on real recorded speech; the rest of the mixes are queued and will be published the same way, including wherever we lose.Reproduce it
The dataset is public. The baseline model is public. The metric is standard, and the normalisation is described above. If you want to check any of this, you can — that is the point of naming everything.
If you make videos in a language the big tools treat as an afterthought, see how we handle yours.
Want subtitles for your video?
Try Harmar.ai for free. The AI turns your Armenian speech into trendy subtitles in a minute or two.