🖐️ Hand-Controlled NUI Cursor System
A touchless Natural User Interface (NUI) desktop controller powered by Google's MediaPipe Tasks API and OpenCV. Designed to map real-time hand gestures and landmark tracking directly to OS cursor navigation and desktop interactions.
🎥 Project Demo Video
NUI Hand Cursor Demo
📋 Architecture & Overview
This system leverages the modern MediaPipe Tasks API (vision.HandLandmarker) running in VIDEO mode combined with OpenCV frame acquisition and PyAutoGUI OS-level automation. By extracting 21 3D hand landmark coordinates per hand in real time, the pipeline computes Euclidean landmark distances and vertical spatial ratios to trigger responsive desktop actions.
🎯 Gesture Interaction Matrix
| Action | Hand Pose & Landmarks | Mathematical Trigger | Visual Indicator |
|---|---|---|---|
| 🖱️ Cursor Movement | Single Index Finger Pointing | Index Tip (Landmark 8) mapped via np.interp |
Green Skeleton & Joints |
| 🖱️ Left Click | Index + Thumb Pinch | Distance(Thumb 4, Index 8) < 30px | Green Banner ("LEFT CLICK") |
| 🖱️ Right Click | Pinky + Thumb Pinch | Distance(Thumb 4, Pinky 20) < 30px | Red Banner ("RIGHT CLICK") |
| 📜 Scroll Up / Down | Index + Middle Fingers Touching | Distance(Index 8, Middle 12) < 30px & Vertical Threshold | Yellow / Magenta Banner |
| ✊ Grab / Drag | Clenched Fist | Avg Distance(Finger Tips to Wrist 0) < 100px | White Banner ("GRAB / DRAG") |
⚙️ Technical Highlights
- Modern Tasks API Integration: Migrated from legacy MediaPipe solutions to the modern
HandLandmarkerTasks API utilizing local asset loading (hand_landmarker.task) optimized for Apple Silicon (ARM64). - Coordinate Mapping & Interpolation: Smoothly maps 2D normalized landmark coordinates to absolute display pixel coordinates using NumPy's 1D linear interpolation (
np.interp). - Dynamic State & Drag Automation: Manages active drag states using
pyautogui.mouseDown()andpyautogui.mouseUp()derived from real-time fist proximity logic. - Fail-Safe & Low Latency Pipeline: Built with OpenCV mirror transformation (
cv2.flip), atomic frame timestamps, and PyAutoGUI safety failsafes to guarantee stable execution.
📊 Model Evaluation & Performance
Core Strengths
- High precision hand skeleton tracking with negligible latency on local webcam feeds.
- Robust multi-gesture discrimination preventing false positive triggers between clicks and movement.
- Efficient CPU/NPU execution on Apple Silicon M-series hardware using pre-compiled task assets.
Limitations & Edge Cases
- Distance Sensitivity: Static pixel threshold boundaries (e.g., 30px click threshold) require consistent user placement relative to camera distance.
- Occlusion Issues: Rapid overlapping finger movements can cause temporary landmark flickering.
- Extreme Lighting: High background glare or extreme low-light environments degrade hand landmark confidence scores below the 0.7 detection threshold.
Validation Test Matrix
| Testing Environment | Cursor Tracking | Pinch Gestures (Click) | Fist Detection (Drag) |
|---|---|---|---|
| Local HD Webcam (Standard Lighting) | Operational | Operational | Operational |
| Extended Distance (>2 Meters) | Operational | Degraded Sensitivity | Operational |
| Low-Illumination Environment | 50% - 60% Accuracy | Degraded | 50% - 60% Accuracy |
🔧 Technical Stack
| Component | Technology / Framework | Application |
|---|---|---|
| Core Engine | Python 3.11 | Primary programming environment |
| Hand Tracking | MediaPipe Tasks API | Real-time 21-landmark 3D hand feature extraction |
| Computer Vision | OpenCV | Frame processing, visual annotations, and mirroring |
| OS Automation | PyAutoGUI | Native mouse positioning, clicking, and scrolling |
| Numerical Mapping | NumPy | Coordinate interpolation and range mapping |
🚀 Engineering Roadmap
- [ ] Implement Exponential Moving Average (EMA) smoothing to eliminate high-frequency cursor jitter
- [ ] Add dynamic distance calibration based on depth/hand-bbox size
- [ ] Incorporate custom multi-finger gestures (e.g., three-finger swipe for workspace switching)
- [ ] Port gesture pipeline to web browser using WebAssembly / MediaPipe JS
- [ ] Integrate custom UI overlay displaying active gesture status and frame metrics
⚠️ Disclaimer & Discretion
Developed strictly for academic research, prototyping, and engineering portfolio demonstration. Designed as a proof-of-concept Natural User Interface (NUI) desktop control solution.