Smart Robotic Arm
Gesture-controlled multi-axis arm with computer vision.
Problem
Remote manipulation systems often lack intuitive control interfaces and suffer from high latency.
Solution
Built a 6-DOF robotic arm controlled via hand gestures using MediaPipe tracking and a real-time WebSocket pipeline.
Approach
Implemented a jitter-free servo control pipeline using PCA9685 and optimized ESP32 firmware for concurrent motor updates.
Outcome
- Achieved near-real-time gesture tracking with <50ms latency.
- Implemented OTA firmware updates for seamless feature delivery.
- Integrated computer vision for precision object manipulation.
Case Study
Overview
A 6-DOF robotic arm controlled in real-time via hand gestures, using MediaPipe for pose estimation and a WebSocket pipeline for low-latency control.
Problem
Most hobbyist robotic arms use joystick or button controls which are unintuitive and slow. Remote manipulation with natural gestures dramatically lowers the learning curve.
Technical Implementation
Host Side (Python)
- OpenCV captures webcam feed at 30fps
- MediaPipe extracts 21 hand landmarks per frame
- Landmark positions mapped to 6 servo angles
- Angles streamed via WebSocket to ESP32
Firmware Side (C++)
- WebSocket server running on ESP32
- PCA9685 PWM driver controls 6 MG996R servos
- Jitter filter smooths incoming angle data
- OTA update endpoint for firmware delivery
Key Achievement
Achieved <50ms gesture-to-movement latency on a local Wi-Fi network, making the control feel natural and immediate.