Hand Gesture-Based Human-Computer Interaction using MediaPipe and OpenCV

Abstract

This study presents the design and implementation of a real-time hand gesture recognition system for directional movement using MediaPipe and OpenCV. The system aims to enhance Human-Computer Interaction (HCI) by recognizing four primary hand gestures—forward, backward, left, and right—based on real-time video input from a standard webcam. The proposed method extracts 21 hand landmarks using MediaPipe, then analyzes landmark displacement to determine the direction of movement. Experiments were conducted under three lighting conditions (bright, moderate, dim) and at three distances (200 cm, 300 cm, and 450 cm). Results show that the system achieved 100% recognition accuracy for all gestures at 200 cm. At 300 cm, accuracy slightly decreased, particularly for backward gestures (down to 77.5%). At 450 cm, performance dropped significantly, with accuracy for some gestures falling below 30%, especially under dim lighting. These findings demonstrate that the proposed system performs reliably at short to medium distances and is sensitive to lighting conditions and user proximity. This research contributes to the development of touchless interfaces for smart environments, presentations, and other interactive applications.

Keywords

Hand gesture; gesture recognition; human-computer interaction; MediaPipe; OpenCV.

Full Text:

PDF

References

V. Gentile, A. Adjorlu, S. Serafin, D. Rocchesso, and S. Sorce, “Touch or touchless?: Evaluating usability of interactive displays for persons with autistic spectrum disorders,” in Proceedings - Pervasive Displays 2019 - 8th ACM International Symposium on Pervasive Displays, PerDis 2019, 2019, pp. 1–7. https://dx.doi.org/10.1145/3321335.3324946

M. Modaberi, “The Role of Gesture-Based Interaction in Improving User Satisfaction for Touchless Interfaces,” Int. J. Adv. Hum. Comput. Interact., vol. 2, no. 2, pp. 20–32, 2024.

Yaseen, O. J. Kwon, J. Kim, J. Lee, and F. Ullah, “Evaluation of Benchmark Datasets and Deep Learning Models with Pre-Trained Weights for Vision-Based Dynamic Hand Gesture Recognition,” Appl. Sci., vol. 15, no. 11, 2025. https://dx.doi.org/10.3390/app15116045

P. Xu, “A Real-time Hand Gesture Recognition and Human-Computer Interaction System,” arXiv:1704.07296, pp. 1–8, 2017.

O. Köpüklü, A. Gunduz, N. Kose, and G. Rigoll, “Real-time hand gesture detection and classification using convolutional neural networks,” Proc. - 14th IEEE Int. Conf. Autom. Face Gesture Recognition, FG 2019, 2019. https://dx.doi.org/10.1109/FG.2019.8756576

E. Fertl, E. Castillo, G. Stettinger, M. P. Cuéllar, and D. P. Morales, “Hand Gesture Recognition on Edge Devices: Sensor Technologies, Algorithms, and Processing Hardware,” Sensors, vol. 25, no. 6, pp. 1–46, 2025. https://dx.doi.org/10.3390/s25061687

A. D. Agustiani, S. M. Putri, P. Hidayatullah, and M. R. Sholahuddin, “Penggunaan MediaPipe untuk Pengenalan Gesture Tangan Real-Time dalam Pengendalian Presentasi [The use of MediaPipe for real-time hand gesture recognition in presentation control],” Media J. Informatics, vol. 16, no. 2, 2024. https://dx.doi.org/10.35194/mji.v16i2.4788

M. Z. Uddin, C. Boletsis, and P. Rudshavn, “Real-Time Norwegian Sign Language Recognition Using MediaPipe and LSTM,” Multimodal Technol. Interact., vol. 9, no. 3, pp. 1–15, 2025. https://dx.doi.org/10.3390/mti9030023.

Y. Astuti and I. D. Ariyanti, “Recognition of hand gestures using image with histogram feature extraction and Euclidean distance classification method,” vol. 13, no. 2, pp. 117–122, 2024. https://dx.doi.org/10.28989/compiler.v13i2.2640

D. Oktaviyanti, A. Nugroho, and A. F. Suni, “Pemanfaatan Hand Tracking untuk Membuat Program Virtual Painter sebagai Alternatif Menggambar Digital,” Petir, vol. 15, no. 2, pp. 287–294, 2022 https://dx.doi.org/10.33322/petir.v15i2.1523

Indriani, M. Harris, and A. S. Agoes, “Applying Hand Gesture Recognition for User Guide Application Using MediaPipe,” Proc. 2nd Int. Semin. Sci. Appl. Technol. (ISSAT 2021), vol. 207, no. Issat, pp. 101–108, 2021. https://dx.doi.org/10.2991/aer.k.211106.017

G. Sánchez-Brizuela, A. Cisnal, E. de la Fuente-López, J. C. Fraile, and J. Pérez-Turiel, “Lightweight real-time hand segmentation leveraging MediaPipe landmark detection,” Virtual Real., vol. 27, no. 4, pp. 3125–3132, 2023. https://dx.doi.org/10.1007/s10055-023-00858-0

J. Abedalrahim, J. Alsayaydeh, T. Lee, C. Jie, R. Bacarra, and B. Ogunshola, “Handwritten text recognition system using Raspberry Pi with OpenCV TensorFlow,” Int. J. Electr. Comput. Eng., vol. 15, no. 2, pp. 2291–2303, 2025. https://dx.doi.org/10.11591/ijece.v15i2.pp2291-2303

K. Patil, S. Ladake, S. Nirgude, and V. Naphade, “Translating Hands Gestures into Text and Speech,” Int. J. Ingenious Res. Invent. Dev., vol. 4, no. 1, pp. 163–172, 2025. https://dx.doi.org/10.5281/zenodo.15168979

P. Abirami, B. Triveni, and D. A. Reddy, “International Journal of Innovative Research in Science Engineering and Technology ( IJIRSET ) Enhanced Communication through Hand Gesture Recognition and Speech Recognition,” Int. J. Innov. Res. Sci. Eng. Technol., vol. 14, no. 4, pp. 9273–9279, 2025. https://dx.doi.org/10.15680/IJIRSET.2025.1404466

http://dx.doi.org/10.28989/avitec.v7i2.3132

Article Metrics

Abstract view: 111 times
Download     : 109   times

This work is licensed under a Creative Commons Attribution 4.0 International License.

Refbacks

• There are currently no refbacks.