Gesture-Controlled 6-DOF Arm with MediaPipe and ESP32

A hand-gesture robotic arm needs four stages: a webcam, a hand tracker, a network link and a servo controller. In my build, Python with OpenCV and MediaPipe reads the hand on a computer, turns it into six servo angles and streams them over a WebSocket to an ESP32. The ESP32 drives six MG996R servos through a PCA9685 board. On a local Wi-Fi network the Smart Robotic Arm reaches less than 50 ms from gesture to movement.

This note walks through the pipeline, why each piece is there, and the two problems that bite first: jitter and power.

The pipeline at a glance

webcam (30 fps) -> OpenCV -> MediaPipe -> 21 landmarks
      -> map to 6 servo angles -> smoothing
      -> WebSocket -> ESP32 -> PCA9685 -> 6 servos

The work is split on purpose. The heavy part, computer vision, runs on the computer. The ESP32 only receives numbers and moves motors.

Host side (Python)

• OpenCV captures the webcam feed at 30 fps.
• MediaPipe extracts 21 hand landmarks per frame.
• Landmark positions are mapped to six servo angles.
• The angles go out over a WebSocket to the ESP32.

Firmware side (C++)

• A WebSocket server runs on the ESP32.
• A PCA9685 PWM driver controls six MG996R servos.
• A jitter filter smooths the incoming angles.
• An OTA update endpoint delivers new firmware without a cable.

Why MediaPipe for the hand

MediaPipe's Hand Landmarker returns 21 landmarks per hand, each with x, y and z coordinates. Its documentation also describes a world-coordinate version in metres, and notes that z is depth relative to the camera, with smaller values closer to it. That gives you a skeleton of the hand every frame without training a model yourself.

The part that takes thought is the mapping. A hand has far more freedom than an arm's joints, so you choose which measurements drive which servo. The ones that are easy to compute from landmarks are positions, distances between two landmarks, and angles between three. A distance such as thumb tip to index tip can become a gripper opening. A hand position in the frame can become base rotation and reach.

I made this easier to try than a video would: the project page has a "Drive the Arm" demo that recreates the pipeline in the browser. MediaPipe reads the hand, inverse kinematics turns it into joint angles, and a smoothing filter keeps the motion calm. The ESP32 and PCA9685 are replaced by a 3D arm on screen.

Why a WebSocket, not HTTP

Gesture data is a stream of small messages in one direction, many times a second. A WebSocket keeps one connection open and lets the host push each new set of angles as soon as it exists. With plain HTTP, each message would carry its own request overhead. A WebSocket server on the ESP32 is also simple to write, and the same Wi-Fi link can carry other messages later.

If you want a telemetry-style design instead, with a broker and many devices, my note on MQTT telemetry for low-power microcontrollers covers that trade-off. For one arm and one controller on the same network, a direct WebSocket is the shorter path.

Why a PCA9685 and not the ESP32's own pins

The PCA9685 board gives 16 channels of 12-bit PWM over I2C. Two things matter for an arm:

• It runs by itself. You send it a value and it keeps producing the pulse. The ESP32 is not tied up generating six timing signals.
• The resolution is fine enough for servos. At the 60 Hz update rate typical for servos, one step of 4096 is about 4 microseconds, since a 60 Hz period is roughly 16.7 ms.

That matters for smooth motion, because the ESP32 can spend its time on the network, the filter and updates, while the PCA9685 holds the pulses steady. If you are new to splitting work like that, my note on FreeRTOS task scheduling on the ESP32 explains how to keep several jobs from starving each other.

Problem one: jitter

Tracking is never perfectly still. Even when your hand rests, landmarks wander by a small amount from frame to frame, and six servos will copy every wobble. The arm twitches and the gears whine.

The fix is a smoothing step between the tracker and the servos. The simplest version is an exponential filter: each new angle is blended with the previous output, so a sudden small change moves the servo only part of the way. I keep this filter in the firmware, so it works no matter which host sends the angles.

The trade-off is delay. A heavier filter is calmer but follows your hand later. A lighter one reacts faster but still shakes. Start light, then raise it until the shaking stops and no further. It also helps to ignore tiny changes below a small threshold, and to clamp each angle to the range its joint can safely reach.

Problem two: power

The servos are the biggest electrical load in the build, and the ESP32 board is the wrong place to power them from. A commonly published MG996R datasheet lists an operating range of 4.8 to 7.2 V. Sellers' listings I checked quote about 2.5 A stall current at 6 V, but the figures vary between sellers and clones, so check the datasheet for the exact servo you buy.

Six servos moving together can pull several amps at the worst moment. So:

• Use a separate 5 to 6 V supply sized for the combined stall current, not just the typical draw.
• Connect the grounds of the servo supply, the PCA9685 and the ESP32 together, or the control signals have no common reference.
• Add a large capacitor across the servo supply to absorb spikes. Values of a few hundred to a thousand microfarads are commonly suggested.
• If the ESP32 resets whenever the arm moves, suspect the power first, not the code.

What I'd do again, and what I'd change

What worked: keeping vision on the computer and the controller dumb. The ESP32 firmware is small, and the OTA endpoint means I can update it without opening the arm. If you are curious how a device can update itself, I wrote up OTA updates from GitHub Releases.

What I would plan for from the start:

• A safe state when the connection drops. If Wi-Fi fails, the arm should hold still or move to a rest pose, not keep chasing the last command.
• Angle limits in the firmware, not only on the host, so a bad message can't drive a joint into its stop.
• A way to test without the camera, such as replaying recorded angles, so you can tune the filter and the power setup before involving your hand.

The less than 50 ms figure is for a local network. A 30 fps camera alone delivers a new frame about every 33 ms, so most of the budget is the camera, not the radio. If you control the arm from far away, expect more delay, and measure it before promising anything.

If you are planning an arm, a vision-controlled build or other connected hardware, see my IoT development services, or get in touch with what you have in mind.