Gesture Recognition untuk Deteksi Bahasa Isyarat BISINDO: Pendekatan Mediapipe dan Random Forest

Authors

  • Salsabila Ayuni Kaffah Universitas ARS Bandung
  • Yudi Ramdhani Universitas Adhirajasa Reswara Sanjaya

DOI:

https://doi.org/10.31328/jointecs.v8i3.4813

Keywords:

Bahasa Isyarat BISINDO, Gangguan Pendengaran, Machine Learning, MediaPipe, Random Forest

Abstract

Gesture Recognition memainkan peran penting dalam memfasilitasi dan meningkatkan aksesibilitas komunikasi bagi individu dengan gangguan pendengaran dan bicara, Namun, dalam menerjemahkan bahasa isyarat yang kompleks menjadi bahasa lisan atau tulisan tetap menjadi tantangan yang signifikan. Berupaya untuk mengatasi hal tersebut, penelitian ini memanfaatkan framework MediaPipe dan algoritma Random Forest Classifier untuk mengklasifikasikan gerakan isyarat berbentuk ungkapan dan kata dalam bahasa isyarat BISINDO. Dengan mempertimbangkan tingkat kesulitan dan kompleksitas gerakan isyarat, 10 label ungkapan/kata dalam BISINDO dipilih dan menghasilkan total 25.000 data yang dipakai pada sistem di penelitian ini. Pendekatan ini melibatkan deteksi bahasa isyarat melalui pengenalan pose, gerakan tangan, dan ekspresi wajah.  Hasil evaluasi menunjukkan algoritma Random Forest mencapai tingkat presisi, recall, F1-score, dan akurasi yang sangat tinggi (99,88%). Selain itu, sistem yang dikembangkan juga menunjukkan kinerja baik dengan rata - rata probabilitas prediksi berkisar antara 0,50 hingga 0,70 untuk prediksi yang benar, meskipun terdapat tantangan dalam membedakan gerakan isyarat yang mirip dan menyebabkan beberapa prediksi memerlukan waktu lebih lama untuk mencapai hasil yang tepat. Dengan hasil yang diperoleh, penelitian ini memberikan kontribusi penting dalam meningkatkan pengenalan bahasa isyarat dan mendorong inklusivitas bagi masyarakat dengan gangguan pendengaran dan bicara. Hal ini juga membuka peluang baru untuk pengembangan lebih lanjut dalam teknologi deteksi bahasa isyarat.

Author Biographies

Salsabila Ayuni Kaffah, Universitas ARS Bandung

Program Studi Teknik Informatika

Yudi Ramdhani, Universitas Adhirajasa Reswara Sanjaya

Program Studi Teknik Informatika

References

World Health Organization, “Deafness and hearing loss,” World Health Organization, 2023. https://www.who.int/news-room/fact-sheets/detail/deafness-and-hearing-loss (accessed Jun. 04, 2023).

S. Rabiah, “Language as a tool for communication and cultural reality discloser,” 2018.

L. R. Nendauni, The Development Of Sign Language: A Synopsis Overview. 2021. doi: 10.13140/RG.2.2.19207.93609.

A. Halder and A. Tayade, “Real-time vernacular sign language recognition using mediapipe and machine learning,” Journal homepage: www. ijrpr. com ISSN, vol. 2582, p. 7421, 2021.

N. P. L. Wedayanti, “Teman Tuli diantara SIBI dan BISINDO,” in Proceedings, 2019, pp. 137–146.

Klobility, “BISINDO dan SIBI: Apa Bedanya?,” Klobility, 2019. https://www.klobility.id/post/perbedaan-bisindo-dan-sibi (accessed Jun. 04, 2023).

C. Janiesch, P. Zschech, and K. Heinrich, “Machine learning and deep learning,” Electronic Markets, vol. 31, no. 3, pp. 685–695, 2021, doi: 10.1007/s12525-021-00475-2.

B. G. Weinstein, “A computer vision for animal ecology,” Journal of Animal Ecology, vol. 87, no. 3, pp. 533–545, May 2018, doi: https://doi.org/10.1111/1365-2656.12780.

A. Ayub Khan, A. Laghari, S. Awan, Lyari, and P. Karachi, “Machine Learning in Computer Vision: A Review,” ICST Transactions on Scalable Information Systems, vol. 8, Apr. 2021, doi: 10.4108/eai.21-4-2021.169418.

M. Nixon and A. Aguado, Feature extraction and image processing for computer vision. Academic press, 2019.

C. E. Kurian, C. Martin, P. J. Deepthi, and K. A. Eldhose, “Air Writing Recognition and Speech Synthesis,” 2018.

J. Kobylarz, J. J. Bird, D. R. Faria, E. P. Ribeiro, and A. Ekárt, “Thumbs up, thumbs down: non-verbal human-robot interaction through real-time EMG classification via inductive and supervised transductive transfer learning,” J Ambient Intell Humaniz Comput, vol. 11, no. 12, pp. 6021–6031, 2020, doi: 10.1007/s12652-020-01852-z.

K. M. Kavana and N. R. Suma, “Recognization of Hand Gestures Using Mediapipe Hands,” International Research Journal of Modernization in Engineering Technology and Science, vol. 4, no. 06, 2022.

D. Denisko and M. M. Hoffman, “Classification and interaction in random forests,” Proceedings of the National Academy of Sciences, vol. 115, no. 8, pp. 1690–1692, 2018.

J. L. Speiser, M. E. Miller, J. Tooze, and E. Ip, “A comparison of random forest variable selection methods for classification prediction modeling,” Expert Syst Appl, vol. 134, pp. 93–101, 2019, doi: https://doi.org/10.1016/j.eswa.2019.05.028.

M. Soni and S. Varma, “Diabetes prediction using machine learning techniques,” International Journal of Engineering Research & Technology (Ijert) Volume, vol. 9, 2020.

I. Hendapratama, I. W. Hamzah, and S. Astuti, “Rancang Bangun Aplikasi Penerjemah SIBI (Sistem Isyarat Bahasa Indonesia) Menggunakan Algoritma Random Forest Classifier,” eProceedings of Engineering, vol. 9, no. 6, 2023.

MediaPipe, “MediaPipe Solutions guide,” 2023. https://developers.google.com/mediapipe/solutions/guide (accessed Jun. 11, 2023).

C. Lugaresi et al., “Mediapipe: A framework for building perception pipelines,” arXiv preprint arXiv:1906.08172, 2019.

K. Goyal, “Indian Sign Language Recognition Using Mediapipe Holistic,” arXiv preprint arXiv:2304.10256, 2023.

Tableau, “Guide To Data Cleaning: Definition, Benefits, Components, And How To Clean Your Data,” Oct. 17, 2021. https://www.tableau.com/learn/articles/what-is-data-cleaning (accessed Jun. 11, 2023).

Mirko Stojiljkovi?, “Split Your Dataset With scikit-learn’s train_test_split(),” Real Python, 2023. https://realpython.com/train-test-split-python-data/#the-importance-of-data-splitting (accessed Jul. 05, 2023).

G. Varoquaux and O. Colliot, “Evaluating Machine Learning Models and Their Diagnostic Value,” in Machine Learning for Brain Disorders, O. Colliot, Ed., New York, NY: Springer US, 2023, pp. 601–630. doi: 10.1007/978-1-0716-3195-9_20.

S. Uddin, I. Haque, H. Lu, M. A. Moni, and E. Gide, “Comparative performance analysis of K-nearest neighbour (KNN) algorithm and its different variants for disease prediction,” Sci Rep, vol. 12, no. 1, p. 6256, 2022, doi: 10.1038/s41598-022-10358-x.

M. M. Saritas and A. Yasar, “Performance analysis of ANN and Naive Bayes classification algorithm for data classification,” International journal of intelligent systems and applications in engineering, vol. 7, no. 2, pp. 88–91, 2019.

B. Charbuty and A. Abdulazeez, “Classification Based on Decision Tree Algorithm for Machine Learning,” Journal of Applied Science and Technology Trends, vol. 2, no. 01, pp. 20–28, Mar. 2021, doi: 10.38094/jastt20165.

H. H. Patel and P. Prajapati, “Study and analysis of decision tree based classification algorithms,” International Journal of Computer Sciences and Engineering, vol. 6, no. 10, pp. 74–78, 2018.

Z. Jun, “The Development and Application of Support Vector Machine,” J Phys Conf Ser, vol. 1748, no. 5, p. 052006, 2021, doi: 10.1088/1742-6596/1748/5/052006.

Y. Ramdhani, R. T. Prasetio, R. Hidayat, and D. P. Alamsyah, “Comparation of SVM Algorithm and Neural Network With Feature Optimization Based on Genetic Algorithm in Determining Immunotherapy Success in Cancer Disease,” in Proceedings of the International Conference on Industrial Engineering and Operations Management Monterrey, 2021, pp. 3142–3149.

Aman Kharwal, “Classification Report in Machine Learning,” The Clever Programmer, Jul. 07, 2021. https://thecleverprogrammer.com/2021/07/07/classification-report-in-machine-learning/ (accessed Jun. 11, 2023).

Aaron Zhu, “Essential Evaluation Metrics for Classification Problems in Machine Learning,” Towards Data Science, Mar. 10, 2023. https://towardsdatascience.com/essential-evaluation-metrics-for-classification-problems-in-machine-learning-69e90665375b (accessed Jun. 11, 2023).

Teemu Kanstrén, “A Look at Precision, Recall, and F1-Score,” towards data science, 2020. https://towardsdatascience.com/a-look-at-precision-recall-and-f1-score-36b5fd0dd3ec (accessed Jul. 14, 2023).

Downloads

Published

2023-09-30

Issue

Section

Articles