Show HN: Live speech-to-text on an ESP32-P4 microcontroller, no cloud

Live English speech recognition on the ESP32-P4. Open vocabulary, streaming, faster than real time, on the microcontroller alone: the board's microphone in, the words on its screen as you speak. No cloud, no neural accelerator; the two RISC-V cores and their PIE SIMD instructions.

The model is Kroko (Banafo), a 65.7M-parameter streaming Zipformer2 transducer, with its weights brought from 71 MB to 26.65 MB in 2 and 4 bits. The weights are on Hugging Face: BoumedineBillal/esp32-pie-asr-kroko-en.

The firmware, firmwares/voice_control, opens on live transcription, then the voice-control demos. The transcriber's numbers were measured with firmwares/live_transcriber, the transcriber alone on the same engine, where the engine's board tests are.

Parameters Weights Runs on Streaming test-clean test-other Speed

esp32-pie-asr 65.7M 2/4-bit, 26.65 MB ESP32-P4 (2 × 400 MHz RISC-V) yes 4.34 % 10.26 % 0.70× real time

Kroko as released (sherpa-onnx) 65.7M int8, 71 MB PC yes 3.94 % 9.17 %

Oído, streaming, with LM 13M int8 ESP32-S3 (240 MHz Xtensa) yes 4.86 % 11.03 % final text 4.1 to 11.1 s after clips of 3.6 to 10.6 s

Oído, whole utterance 13M int8 ESP32-S3 no 3.70 % 8.23 % 1.97× real time (one core)

Moonshine Tiny 27.1M float CPU no 4.52 % 11.71 %

Whisper tiny.en 37.8M float CPU no 5.66 % 15.45 %

ESP-SR MultiNet

ESP32-S3

a fixed list of up to 200 commands, not open vocabulary

esp32-pie-asr and Kroko as released: measured here on all 2620 / 2939 clips with Whisper's English text normalizer (Kroko with sherpa-onnx 1.13.8, the same beam width, 2 s of silence after each clip so its last chunk is decoded). The others: as their authors publish them.

The 2/4-bit weights cost about half a point; the integer arithmetic on the chip costs nothing.

A text file on the SD card lists words and sentences to listen for; a word the model spells differently gets its spoken forms (ESP32 = ESP thirty two). Each one is switched on or off on the screen. On 140 text-to-speech clips (35 sentences, 4 voices) with the default list:

The firmware's second page has six demos in which spoken orders drive a device simulated on the screen: a robot vacuum, a smart home, a car cabin, a factory line, a kitchen, a presenter. No language model: each demo is a table of word rules (an order's verb, then its target; numbers, rooms, "not the bedroom"), and the screen shows which words fired each action. Each demo has its own phrase list.

On 188 text-to-speech clips per demo (25 orders and 22 near misses, ordinary sentences with the demo's words, 4 voices):

The lists were chosen on these clips; every miss is a word heard wrong. The board gives the PC's actions on every clip. From the end of an order to its action: median 1.84 s on the vacuum's page, 1.97 s on the factory's, at most 2.57 s. With a demo's page drawing, the engine stays under real time (at most 0.81×).

Hardware: an ESP32-P4 of chip revision v3 (ESP32-P4X) with 32 MB PSRAM, a microphone, a screen, a microSD card. Made and measured on Espressif's ESP32-P4X-Function-EV-Board (ES8311 microphone, 1024 × 600 MIPI-DSI panel).

The weights (26.65 MB) do not fit the board's 16 MB flash, so they live on a microSD card: at boot the firmware reads them from the card into the 32 MB PSRAM (2.9 s) and runs from there.

git clone https://github.com/BoumedineBillal/esp32-pie-asr cd esp32-pie-asr python models/download.py --card <the microSD card's folder> # downloads, checks, copies

Put the card in the board's microSD slot.

Flash the firmware, from the browser (Chrome or Edge, no tools to install): boumedinebillal.github.io/esp32-pie-asr, or with ESP-IDF v6.1: cd firmwares/voice_control idf.py set-target esp32p4 idf.py -p PORT flash

At the first boot the firmware also writes its phrase lists on the card: phrases.txt for the transcriber and one per demo (vacuum_phrases.txt, smarthome_phrases.txt, ...). To change a list, edit its file on a PC: one word or sentence per line, shown = spoken | spoken for a word the model spells differently, # for a note; the screen shows up to 10 words and 10 sentences.

firmwares/live_transcriber (the transcriber alone) builds and flashes the same way.

The debug build adds a link to a PC over the board's USB High-Speed port (commands, the log, the words, screenshots) for the board tests (debug/tests/run_all.py, Windows):

idf.py -B build_debug -D SDKCONFIG=build_debug/sdkconfig
-D SDKCONFIG_DEFAULTS="sdkconfig.defaults;sdkconfig.debug" -p PORT flash

Inside

How the model fits and runs, in docs/design.md:

cmake -S pieasr/host -B build/host -DPIEASR_WEIGHTS=$PWD/models/kroko-en-q24-v1.bin cmake --build build/host && ctest --test-dir build/host # unit tests and the 8 golden clips python tools/eval/get_librispeech.py --set test-clean # 2620 clips; --set test-other: 2939 bash tools/eval/run_wer.sh data/test-clean int+beam4 # the chip's configuration, on the PC bash pieasr/tests/target/run.sh # every PIE kernel against its C reference, on the board

Next

Support for the ESP32-S31, which has the same PIE 2.2 SIMD instructions as the ESP32-P4X.

Arabic: a model for Modern Standard Arabic.

Credits

Kroko by Banafo: the model (community license, CC-BY-SA).

Cactus Quants by Cactus Compute: the weight format (Hadamard-rotated groups of 128, Lloyd-Max levels).

LibriSpeech (CC BY 4.0): the 8 test clips in pieasr/tests/data and the WER sets.

Piper and its voices: the text-to-speech test clips (made by the tests' scripts, not in the repo).

Espressif: ESP-IDF, and the ES8311 and EK79007 register values (Apache-2.0).

Inter (SIL OFL 1.1) and stb_truetype (public domain or MIT).

License

The code: Apache-2.0 (LICENSE). The weights: CC-BY-SA 4.0, as Kroko's.

Boumedine Billal: GitHub · Hugging Face · LinkedIn · boumedinebillal@gmail.com