Whisper live transcription.

Why Whisper is not a streaming model

The input size is fixed at 30 seconds. Shorter clips are padded with zeros, and longer audio is cut into 30 second windows that are transcribed one after another. In OpenAI's repo the constant is CHUNK_LENGTH = 30, with a comment counting 480000 samples per chunk. Hugging Face's card for large-v3 says the model has a receptive field of 30 seconds.

OpenAI's model card says the models cannot be used for real-time transcription out of the box, and that others may be able to build near real time applications on top of them.

Sliding windows with local agreement

The idea that made this work is LocalAgreement. Feed Whisper a growing buffer, run it again as audio arrives, and confirm only the prefix that two consecutive runs agree on. Confirmed text is not rewritten. The 2023 whisper_streaming paper reports 3.3 seconds of latency on a long form test set.

Read its README first. It says the project is becoming outdated and is being replaced by SimulStreaming from the same author. SimulStreaming's README calls it about 5 times faster, and it uses an attention guided policy called AlignAtt instead. It recommends a GPU with at least 10 GB of VRAM for large-v3 and says a CPU would be too slow for real time.

Both commands below simulate real time from a 16 kHz mono wav file. In whisper_streaming's source the default faster-whisper backend loads the model on CUDA, so it needs an NVIDIA GPU. On Apple silicon use the mlx-whisper backend. With no model flag, whisper_streaming downloads large-v2 and SimulStreaming downloads large-v3. Each repo also ships a TCP server you pipe raw microphone audio into.

# make a 16 kHz mono wav from any recording
ffmpeg -i input.mp3 -ar 16000 -ac 1 -c:a pcm_s16le audio.wav

# whisper_streaming, NVIDIA GPU
git clone https://github.com/ufal/whisper_streaming
cd whisper_streaming
pip install librosa soundfile faster-whisper torch torchaudio
python3 whisper_online.py ../audio.wav --language en --min-chunk-size 1 --vac > out.txt

# whisper_streaming on Apple silicon
pip install librosa soundfile mlx-whisper torch torchaudio
python3 whisper_online.py ../audio.wav --backend mlx-whisper --language en --min-chunk-size 1 --vac > out.txt

# SimulStreaming, the successor
cd ..
git clone https://github.com/ufal/SimulStreaming
cd SimulStreaming
pip install -r requirements_whisper.txt
python3 simulstreaming_whisper.py ../audio.wav --language en --task transcribe --vac

whisper.cpp stream, the quickest thing to try

whisper.cpp ships a binary called whisper-stream. Its README calls it a naive example: it samples the audio every half second and transcribes continuously, rewriting one rolling line in the terminal.

Setting --step to 0 switches it to sliding window mode. A basic voice activity detector waits for speech, and at silence it transcribes the last --length milliseconds and prints a timestamped block meant to be parsed. The README suggests about 0.6 for -vth.

git clone https://github.com/ggml-org/whisper.cpp.git
cd whisper.cpp
brew install sdl2   # Debian and Ubuntu: sudo apt-get install libsdl2-dev
cmake -B build -DWHISPER_SDL2=ON
cmake --build build --config Release
sh ./models/download-ggml-model.sh base.en
./build/bin/whisper-stream -m ./models/ggml-base.en.bin -t 6 --step 0 --length 30000 -vth 0.6

WhisperLive, a server your apps can talk to

WhisperLive from Collabora is a WebSocket server with Python and browser clients. Its README calls it a nearly live implementation of Whisper. One protocol fronts three backends, faster-whisper, TensorRT-LLM and OpenVINO, and a client can send a microphone, a file, an RTSP stream or an HLS stream.

The README also documents word level timestamps, hotwords for custom vocabulary, optional diarization, and a use_vad switch for server side detection. The client line below streams a sample file from the repo. Leave --files out to use the microphone. The README installs into a Python 3.12 virtual environment.

git clone https://github.com/collabora/WhisperLive
cd WhisperLive
bash scripts/setup.sh   # installs PortAudio for the client, needs root on Linux
pip install whisper-live
python3 run_server.py --port 9090 --backend faster_whisper --max_clients 4

# in a second shell
python3 run_client.py --files assets/jfk.flac

What every one of these costs you

• Latency against accuracy. A shorter window prints sooner and gets more wrong.
• Text that rewrites itself. Rolling windows revise the line you are reading. Local agreement stops that by waiting for two runs to agree.
• Repeats and invented words. OpenAI's model card says the sequence to sequence design makes the models prone to repetitive text, and that predictions may include text that was not spoken. It says beam search and temperature scheduling help with repetition, but not perfectly.
• Hardware. The table in OpenAI's README lists about 10 GB of VRAM for the 1550 million parameter large models, and about 6 GB for turbo at 809 million.

The other route: run a fast model after every pause

• Pick Whisper for a language those 25 do not cover, which is most of the world.
• Pick Whisper for recordings. Fraze is live only, and Whisper on a whole file is free and sees the full context.
• Pick Whisper to turn speech into English in one pass, with --task translate. Its README notes turbo is not trained for translation.

If a model runs far faster than real time, you do not need a streaming policy. Put a voice activity detector in front, wait for the speaker to stop, then transcribe the segment that just ended. The text is written once and never revised.

That is how Fraze works, and Fraze does not run Whisper. Its engine is NVIDIA's Parakeet TDT 0.6B v3, a 600 million parameter model under CC BY 4.0 covering 25 European languages, run through sherpa-onnx as int8 ONNX files with Silero VAD marking the boundaries. Captions land about half a second to a second after a pause.

Accuracy is a trade, not a win. These are our own runs, not an independent test. Word error rate in percent, lower is better: LibriSpeech clean, Parakeet 2.15 against Whisper large-v3-turbo 1.93. Earnings-22 conference calls, Parakeet 11.20 against 12.35, but that Whisper run went through WhisperKit, a Core ML port, and the benchmark repository notes that the reference PyTorch build scores about 11.1 there, level with Parakeet. On FLEURS read speech, 200 clips per language, with a hosted Whisper large-v3-turbo, Whisper leads in all four languages both models cover: German 5.33 against 6.21, Spanish 5.68 against 6.05, French 6.82 against 7.83, Russian 7.48 against 9.18. In Fraze's own engine Parakeet ran at about 0.03 times real time on an Apple silicon Mac mini, which is what makes the pause approach work. Whisper also reaches further: its model card says the non-English training data alone spans 98 languages.

Questions