Four sets of vendor claims, and why none of them can be ranked

Some of the comparison needs no benchmark at all, because it is fact-checkable from vendor documentation. Meta’s streaming model runs at $0.18 an hour of audio, OpenAI’s file transcription at $0.27, and Microsoft’s batch model at $0.10 as a launch offer with no standard rate published. All three require the audio to leave the machine and none of them ship weights. Running locally costs nothing an hour, which is the whole argument.

OpenAI is worth separating out, because you cannot buy the capability set as one product. A transcript of a file, live audio, knowing who spoke, and word timestamps are four different models at four prices, and their own guidance tells you which to use for which.

The accuracy numbers are where it falls apart. Meta reports 3.1% word error against a private index nobody can reproduce. Microsoft reports 5.2% averaged across sixty languages, and 3.4% across the top twenty five, so the choice of language set moves that figure by 1.8 points on its own. OpenAI publishes nothing, which makes any claim about its accuracy an invention. Four corpora, four methods, no row you can sort.

Measured locally on one Apple M4, CPU only, across the full LibriSpeech dev-clean set: 4.40% with one recogniser and 6.81% with the shipped default. The honest reading is that both sit in the four to seven percent range on easy read speech, and not that the lower one wins, because that model was trained on LibriSpeech and its gains concentrate in proper nouns from the books the corpus is read from. Benchmarking a model on its own training distribution flatters it.

Diarization discovers the speaker count rather than taking it as configuration, and was exact from two speakers up to twenty four, with confusion between 3.1% and 7.6%. Three people through one laptop microphone returns two. That failure has a measured cause rather than a tuning fix: two of the voices sat at 0.54 cross-similarity against 0.55 within, so no threshold separates them. The shared room and microphone signature swamps the individual one.

A real head-to-head is tractable and costs about three dollars. The hard part, normalising case, punctuation and numbers so polished vendor output scores fairly against bare references, is already solved. What is missing is an adapter that walks the same files and posts them to each endpoint.