Computer vision

Hand gesture control

A touchless mouse, slide remote and air-drawing whiteboard, driven by real-time hand tracking and a classifier trained on your own recordings.

The problem

Controlling a computer with hand gestures sounds like a classification problem, but most of the work is elsewhere. A cursor has to be steady without lagging, and a single misread frame must not fire a click. I built a gesture mouse, a slide remote and an air-drawing whiteboard on top of webcam hand tracking.

What I built

  1. MediaPipe Hand Landmarker gives 21 3D landmarks per frame. They are shifted to the wrist, left hands are mirrored and everything is scaled by palm size, so a classifier sees shape rather than position.
  2. Six static gestures are recognised by a rule-based baseline or by an MLP trained on the user's own recordings, with a session-level train/test split.
  3. A gesture must hold for several frames before it becomes active, and low-confidence frames return no gesture.
  4. The cursor follows the index knuckle through a One Euro filter, and a mode controller switches between mouse, slides and drawing.

Results

Per-frame latency from gesture-benchmark on an Apple M5 CPU; gesture accuracy is not measured yet.
StageTime per frame
Hand tracking (MediaPipe)about 8 ms
Gesture classificationunder 0.1 ms
Gesture accuracynot measured yet

What the results show

  • The frame rate is limited by the camera, not by tracking or classification.
  • Following the knuckle instead of the fingertip matters for clicks: pinching moves the fingertip, while the knuckle stays where you aimed.
  • The One Euro filter smooths heavily when the hand is still and lightly when it moves fast, which removes jitter without adding visible lag.
  • Consecutive frames are near-duplicates, so a random split would leak them into the test set. Holding out each gesture's latest session tests a new recording instead.

Limitations

  • No accuracy numbers until a gesture dataset is recorded, and a model trained on one person's hand may not transfer to others.
  • Static, one-hand gestures only. Dynamic or two-hand gestures would need a temporal model over landmark sequences.

Full results, commands and figures are in the repository README.