A Machine Learning-Based Ambiguous Alphabet Recognition for Indonesian Sign Language System (SIBI)

Yoan Purbolingga; Ahmad Ridwan; Dila Marta Putri

doi:10.31154/cogito.v11i1.816.1-14

Authors

Yoan Purbolingga Institut Teknologi Bisnis Riau
Ahmad Ridwan Universitas AMIKOM Yogyakarta
Dila Marta Putri Institut Teknologi Bisnis Riau

DOI:

https://doi.org/10.31154/cogito.v11i1.816.1-14

Keywords:

Sign Language, Image Processing, SIBI, Machine Learning, Chain Code

Abstract

One of the communication problems in deaf people is the inhibition of verbal communication. This is due to the limited hearing function which has an impact on the imperfection of language sound reception. To communicate with deaf people, extraordinary communication is needed so that the meaning of the conversation can be conveyed properly. Sign language is the main communication medium for deaf people. However, in the use of sign language, there are ambiguous letters, namely “D “,“E“,“M“,“N“,“R“, “S“, and “U“. This research uses the chain code method to identify and reconstruct the shape of hand gesture objects. Then, to solve the problem of ambiguity of alphabet letters, an artificial intelligence method, namely K-Nearest Neighbors (K-NN), is used. The sample used consists of 350 real-time images with variations in object recognition accuracy. Based on the research using chain code and K-NN classification method, it can be concluded that the recognition of ambiguous letters in sign language has 245 training data for K-NN which has 88.76% accuracy, and 105 test data with 90% accuracy. This test data is divided into seven letters: “D“, “E”, “M”, “R” and “U” at 100%, and “N” and “S” at 98.88%.

Author Biography

Ahmad Ridwan, Universitas AMIKOM Yogyakarta

Department Informatics, Faculty of Computer Science

References

F. Fitriyani, L. Q. Ainii, R. Jannah, and S. Maryam, “Analysis of Sign Language Skills in Improving Communication and Learning for Deaf Children,” Continous Educ. J. Sci. Res., vol. 5, no. 1, pp. 30–39, 2024, doi: 10.51178/ce.v5i1.1757.

K. Wangchuk, P. Riyamongkol, and R. Waranusast, “Real-time Bhutanese Sign Language digits recognition system using Convolutional Neural Network,” ICT Express, vol. 7, no. 2, pp. 215–220, 2021, doi: 10.1016/j.icte.2020.08.002.

M. J. Cheok, Z. Omar, and M. H. Jaward, “A review of hand gesture and sign language recognition techniques,” Int. J. Mach. Learn. Cybern., vol. 10, no. 1, pp. 131–153, 2019, doi: 10.1007/s13042-017-0705-5.

S. Maullasari, “The role of social support to increase the Confidence of the Deaf in the Difabel Study Council,” J. Adv. Guid. Couns., vol. 3, no. 2, pp. 94–105, 2022, doi: 10.21580/jagc.0.0.0.5896.

A. Harditya, “Indonesian Sign Language (BISINDO) As Means to Visualize Basic Graphic Shapes Using Teachable Machine,” in Proceedings of the International Conference of Innovation in Media and Visual Design (IMDES 2020), 2020, pp. 1–7. doi: 10.2991/assehr.k.201202.045.

S. Syamsuddin, T. Pristiwaluyo, W. A. Saleh, and Z. Zulfitrah, “Development of Indonesian Sign Language System (SIBI) Dictionary Application for Students with Special Needs,” Al-Ishlah J. Pendidik., vol. 15, no. 3, pp. 3132–3143, 2023, doi: 10.35445/alishlah.v15i3.3529.

G. Latif, J. Alghazo, N. Mohammad, and R. Alghazo, “Communicating with the Deaf and Hard of Hearing through Automatic Arabic Sign Language Translator,” in Journal of Physics: Conference Series, 2021. doi: 10.1088/1742-6596/1962/1/012055.

S. Shivashankara and S. Srinath, “American Sign Language Recognition System: An Optimal Approach,” Int. J. Image, Graph. Signal Process., vol. 10, no. 8, pp. 18–30, 2018, doi: 10.5815/ijigsp.2018.08.03.

N. B. Ibrahim, M. M. Selim, and H. H. Zayed, “An Automatic Arabic Sign Language Recognition System (ArSLRS),” J. King Saud Univ. - Comput. Inf. Sci., vol. 30, no. 4, pp. 470–477, 2018, doi: 10.1016/j.jksuci.2017.09.007.

I. D. M. B. A. Darmawan et al., “Advancing Total Communication in SIBI: A Proposed Conceptual Framework for Sign Language Translation,” in 2023 International Conference on Smart-Green Technology in Electrical and Information Systems (ICSGTEIS), 2023, pp. 23–28. doi: 10.1109/ICSGTEIS60500.2023.10424020.

A. Aljabar and Suharjito, “BISINDO (Bahasa isyarat indonesia) sign language recognition using CNN and LSTM,” in Advances in Science, Technology and Engineering Systems, 2020, pp. 282–287. doi: 10.25046/AJ050535.

A. D. Saputra, J. Jayanta, and A. B. Pangaribuan, “Klasifikasi Alfabet Bahasa Isyarat Indonesia (BISINDO) Dengan Metode Template Matching dan K-Nearest Neighbors (KNN),” in Seminar Nasional Mahasiswa Ilmu Komputer dan Aplikasinya (SENAMIKA), 2020.

R. K. Pathan, M. Biswas, S. Yasmin, M. U. Khandaker, M. Salman, and A. A. F. Youssef, “Sign language recognition using the fusion of image and hand landmarks through multi-headed convolutional neural network,” Sci. Rep., vol. 13, no. 1, pp. 1–11, 2023, doi: 10.1038/s41598-023-43852-x.

P. W. Aditama, P. S. U. Putra, I. M. M. Yusa, and I. N. T. A. Putra, “Designing augmented reality sibi sign language as a learning media,” in Journal of Physics: Conference Series, 2021. doi: 10.1088/1742-6596/1810/1/012038.

K. Bantupalli and Y. Xie, “American Sign Language Recognition using Deep Learning and Computer Vision,” in Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018, IEEE, 2018, pp. 4896–4899. doi: 10.1109/BigData.2018.8622141.

A. Ridwan, Y. Purbolingga, and H. Hanisah, “Utilizing Convolutional Neural Network for Learning Web-Based Braille Letter Classification System,” J. Comput. Networks, Archit. High Perform. Comput., vol. 6, no. 1, Jan. 2024, doi: 10.47709/cnahpc.v6i1.3386.

P. Yugopuspito, I. M. Murwantara, and J. Sean, “Mobile Sign Language Recognition for Bahasa Indonesia using Convolutional Neural Network,” in Proceedings of the 16th International Conference on Advances in Mobile Computing and Multimedia, in MoMM2018. New York, NY, USA: Association for Computing Machinery, 2018, pp. 84–91. doi: 10.1145/3282353.3282356.

R. Ridwang, S. Syafaruddin, A. A. Ilham, and I. Nurtanio, “Indonesian Sign Language Letter Interpreter Application Using Leap Motion Control based on Naïve Bayes Classifier,” in IOP Conference Series: Materials Science and Engineering, 2019. doi: 10.1088/1757-899X/676/1/012012.

A. Nur and H. Nugroho, “Deteksi Fitur Huruf Sistem Isyarat Bahasa Indonesia menggunakan Metode Chain Code,” POSITIF J. Sist. dan Teknol. Inf., vol. 8, no. 1, pp. 36–40, 2022, doi: 10.31961/positif.v8i1.1133.

H. M. Putri, F. Fadlisyah, and W. Fuadi, “Pendeteksian Bahasa Isyarat Indonesia Secara Real-Time Menggunakan Long Short-Term Memory (LSTM),” J. Teknol. Terap. Sains 4.0, vol. 3, no. 1, p. 663, 2022, doi: 10.29103/tts.v3i1.6853.

I. Suyudi, S. Sudadio, and S. Suherman, “Pengenalan Bahasa Isyarat Indonesia menggunakan Mediapipe dengan Model Random Forest dan Multinomial Logistic Regression,” J. Ilmu Siber dan Teknol. Digit., vol. 1, no. 1, pp. 65–80, 2023, doi: 10.35912/jisted.v1i1.1899.

N. A. Hasma, F. Arnia, R. Muharar, and M. K. Muchamad, “Pengenalan Gerakan Isyarat Bahasa Indonesia Menggunakan Algoritma Surf Dan K-Nearest Neighbor,” J. Komputer, Inf. Teknol. dan Elektro, vol. 7, no. 1, p. 2022, 2022, [Online]. Available: https://jurnal.unsyiah.ac.id/kitektro/article/view/23262

M. E. Al Rivan, H. Irsyad, K. Kevin, and A. T. Narta, “Implementasi LDA pada fitur HOG untuk Klasifikasi ASL Menggunakan K-NN,” JATISI (Jurnal Tek. Inform. dan Sist. Informasi), vol. 7, no. 2, pp. 214–225, 2020, doi: 10.35957/jatisi.v7i2.286.

S. V Sonekar, H. Dhoke, V. Mate, S. Dhewle, and M. Patil, “Real-Time Sign Language Identification using KNN: A Machine Learning Approach,” in 2023 11th International Conference on Emerging Trends in Engineering & Technology - Signal and Information Processing (ICETET - SIP), 2023, pp. 1–4. doi: 10.1109/ICETET-SIP58143.2023.10151523.

A. Sultan, W. Makram, M. Kayed, and A. A. Ali, “Sign language identification and recognition : A comparative study,” Open Comput. Sci., vol. 12, no. 1, pp. 191–210, 2022.