Wake-Word Detection on an ESP32-S3 with TensorFlow Lite Micro

You can detect a wake word on an ESP32-S3 without any cloud service. A microphone feeds audio into a feature step, a small TensorFlow Lite Micro model decides whether the keyword was just spoken, and your firmware reacts. I did this for my SmartPot plant companion, where the wake word is recognised in under 100 ms and the full path from wake word to spoken reply stays under 200 ms, all on the device.

This is the build log: the shape of the pipeline, the decisions that matter, and the problems to plan for before you start. I've kept to what I built and to published facts about the tools, and I link to those.

What the SmartPot does with a wake word

The SmartPot is a voice-controlled plant assistant. It also reads soil moisture, temperature, humidity and light, so the wake word is only one job on the board. The voice path works like this:

1. The wake word is detected.
2. A 1.5 second audio buffer is captured.
3. The command is classified into one of more than 20 plant-care intents.
4. A reply is played through a DFPlayer Mini.

Everything after step 1 runs locally too. That is the point of the build: a plant monitor that still talks when the Wi-Fi is down.

The pipeline, stage by stage

microphone -> audio buffer -> feature extraction
           -> TensorFlow Lite Micro model -> score
           -> wake word? -> capture command -> intent -> reply

1. Audio in. A digital microphone gives a steady stream of samples. Keyword models are usually trained on 16 kHz audio, so capture at that rate and keep a rolling buffer rather than recording on demand. A wake word can start at any moment, and a model that only listens after a button press is not a wake-word detector.

2. Features, not raw audio. The model doesn't read waveforms. In TensorFlow's standard micro_speech example, one second of audio becomes a spectrogram of 49 slices with 40 frequency values each, which you can see in the example's settings header. The firmware has to compute those numbers exactly the way the training script did. If the two disagree, the model still runs and gives confident, wrong answers.

3. The model. Typically a small network quantised to 8-bit integers, converted with the TensorFlow Lite converter and linked into the firmware as a byte array. I covered the loading steps and the tensor arena in running TensorFlow Lite Micro on the ESP32-S3.

4. The decision. The model outputs a score per class. You don't act on a single frame. Wait for the score to stay high across a few consecutive windows, or you will wake up on a cough.

Decisions to make first

ESP-SR is a good choice if one of its supported wake words does the job. The SmartPot uses a TensorFlow Lite model and its own intent router for the 20+ plant-care commands. Neither route is wrong. If you only need the device to wake up and listen, ESP-SR will get you there with less work.

Problems to plan for

Audio and everything else share one chip. The SmartPot runs four FreeRTOS tasks on one ESP32-S3: audio, sensor polling, MQTT telemetry and a web server. The audio task has to keep up with the microphone with no gaps, so it gets its own priority, and slow work like network calls must never block it. I wrote about this in FreeRTOS task scheduling on the ESP32.

Memory is the budget. The model, its tensor arena, the audio buffer and the feature buffer all compete with the Wi-Fi stack and the web server. Decide early what lives in internal RAM and what goes to PSRAM, and check free heap while the whole firmware is running, not just the voice code on its own.

False wakes and missed wakes trade off. Raise the threshold and the device ignores your friend across the room. Lower it and it answers the TV. There is no setting that fixes both, so test in the room where the device will actually sit: with fans, other voices, and a plant being watered nearby.

Train on the microphone you will ship. A model trained on clean recordings can struggle with the frequency response and noise of a cheap board microphone. Record some of your training data through the real hardware.

Measure latency end to end. Time from the wake word to the start of the spoken reply, not just the model's inference time. The reply path, which in my build includes the DFPlayer Mini, counts as much as the model does.

Keep settings across power cycles. Volume and preferences are saved to NVS flash on the SmartPot, so the device doesn't reset to defaults every time it is unplugged.

What I'd recommend

• Start with the smallest model that detects one wake word, and add intents after that works.
• Keep the voice path fully local, so it never depends on Wi-Fi.
• Write the feature code once and share it between training and firmware tests.
• Keep the audio task isolated, and let sensors, MQTT and the dashboard fit around it.
• Log scores from real use, so the threshold comes from data and not from guessing.

If you are building a device that needs to listen and respond on its own, see my AI development services or get in touch with a short description of what it should do.