Wireless Gesture Controller (ESP32 + MediaPipe)

Developer: Mudit Bhalla (25BCE0370)
Institution: VIT Vellore

An end-to-end hardware-software integration project that transforms a standard webcam into a wireless gesture-control interface. Utilizing Google's MediaPipe framework and OpenCV, the application detects hand landmarks in real time, interprets the number of extended fingers, and transmits control signals across a local Wi-Fi network to an ESP32 microcontroller via low-latency UDP Sockets.

Technology Stack

• Languages: Python 3, C++ (Arduino Framework)
• Computer Vision: OpenCV, MediaPipe
• Networking: UDP Socket Programming (socket)
• Hardware Target: ESP32 Microcontroller

Project Architecture

1. Frame Capture: OpenCV pulls real-time frames from the local webcam.
2. Landmark Extraction: MediaPipe maps 21 dynamic coordinate points across the user's hand.
3. Gesture Logic: Algorithms evaluate spatial geometric relationships to determine finger extension counts (0–5).
4. Network Payload: A localized UDP socket serializes and broadcasts data blocks over the network.
5. Hardware Execution: The ESP32 parses incoming byte packets to execute real-time physical actions.

---

---

Getting Started

Prerequisites

Ensure your machine runs Python 3.x and has network access to your ESP32's local gateway.

Installation & Setup

1. Clone the repository:

   git clone [https://github.com/YOUR_GITHUB_USERNAME/esp32-gesture-control.git](https://github.com/YOUR_GITHUB_USERNAME/esp32-gesture-control.git)
   cd esp32-gesture-control

2. Install the required Python dependencies:

   python3 -m pip install -r python_hand_tracker/requirements.txt

3. Update the execution script: Open python_hand_tracker/roboVIT_night2.py and replace the network configurations with your hardware layout parameters:

   ESP32_IP = "YOUR_ESP32_IP_ADDRESS"
   ESP32_PORT = 1234

4. Running the Application

   python3 python_hand_tracker/roboVIT_night2.py

   Press 'q' on your keyboard while focusing on the camera screen to safely shut down streams and clean sockets.

5. Troubleshooting Are you developing on a Mac and hitting camera crashes or environment path bugs? Check out our step-by-step resolution roadmap in TROUBLESHOOTING.md.

6. The requirements.txt File Inside your python_hand_tracker/ directory, create a file named requirements.txt containing your script dependencies:

   opencv-python
   mediapipe