Sentiment Classification of IT Service Feedback via TF-IDF

Samidi Samidi; Devy  Fatmawati

doi:10.31154/cogito.v10i2.701.403-417

Authors

Samidi Samidi Universitas Budi Luhur
Devy Fatmawati Universitas Budi Luhur

DOI:

https://doi.org/10.31154/cogito.v10i2.701.403-417

Keywords:

Classification, Feedback, Naïve Bayes, SVM, K-NN

Abstract

Handling user complaints and feedback is a key strategy of Pusintek, the Ministry of Finance of the Republic of Indonesia, to enhance user satisfaction. The challenge faced is the difficulty in accurately analyzing feedback due to differences in comments and categories chosen by users, which requires manual category correction. This study aims to automate feedback comment categorization using classification algorithms. Specifically, Naïve Bayes, Support Vector Machine (SVM), and K-Nearest Neighbors (K-NN) algorithms were applied to 11,108 user feedback records. The CRISP-DM framework was used, with dataset preparation involving sentiment analysis techniques (cleansing, case folding, normalization, filtering, and tokenization) and Term Frequency-Inverse Document Frequency (TF-IDF) weighting. Accuracy values for each algorithm were evaluated. Results show that the SVM algorithm performed the best, achieving an accuracy of 94.10% and consistently delivering the highest precision, recall, and f1-score across all sentiment categories. This research contributes to the development of an automatic feedback classification system that improves categorization accuracy, minimizes manual intervention, and optimizes user feedback analysis. It is expected to enrich the understanding of text classification and natural language processing techniques and open up opportunities for further research.

References

D. Istanbulluoglu and E. Sakman, "Successful complaint handling on social media predicts increased repurchase intention: The roles of trust in company and propensity to trust," Eur. Manag. J., vol. 42, no. 1, pp. 11–22, 2024.

K. Lou Musu, C. Suryawati, and H. Warsono, "Analisis Sistem Penanganan Komplain di Rumah Sakit Permata Medika Semarang," 2020.

F. V. Sari and A. Wibowo, "Analisis Sentimen Pelanggan Toko Online JD.ID Menggunakan Metode Naïve Bayes Classifier Berbasis Konversi Ikon Emosi," J. Simetris, vol. 10, no. 2, pp. 681–686, 2019.

M. Birjali, M. Kasri, and A. Beni-Hssane, "A comprehensive survey on sentiment analysis: Approaches, challenges and trends," Knowl.-Based Syst., vol. 226, p. 107134, 2021.

C. Zong, R. Xia, and J. Zhang, Text Data Mining, vol. 711, Singapore: Springer, 2021.

M. E. Basiri, M. Abdar, M. A. Cifci, S. Nemati, and U. R. Acharya, "A novel method for sentiment classification of drug reviews using fusion of deep and machine learning techniques," Knowl.-Based Syst., vol. 198, p. 105949, 2020.

M. Wongkar and A. Angdresey, "Sentiment analysis using Naive Bayes Algorithm of the data crawler: Twitter," in 2019 Fourth Int. Conf. Informatics Comput. (ICIC), 2019, pp. 1–6, doi: 10.1109/ICIC47613.2019.8985837.

S. H. Kusumahadi, H. Junaedi, and J. Santoso, "Klasifikasi Helpdesk Menggunakan Metode Support Vector Machine," J. Inform., vol. 4, no. 1, pp. 54–60, Jan. 2019, doi: 10.30591/jpit.v4i1.1125.

E. Indrayuni, A. Nurhadi, and D. A. Kristiyanti, "Implementasi Algoritma Naive Bayes, Support Vector Machine, dan K-Nearest Neighbors untuk Analisa Sentimen Aplikasi Halodoc," Faktor Exacta, vol. 14, no. 2, p. 64, Aug. 2021, doi: 10.30998/faktorexacta.v14i2.9697.

M. Azhar, N. Hafidz, B. Rudianto, and W. Gata, "Marketplace Sentiment Analysis Using Naive Bayes And Support Vector Machine," PIKSEL, vol. 8, no. 2, pp. 91–100, Sep. 2020, doi: 10.33558/piksel.v8i2.2272.

Hermanto, A. Mustopa, and A. Y. Kuntoro, "Algoritma Klasifikasi Naïve Bayes dan Support Vector Machine Dalam Layanan Komplain Mahasiswa," 2020. [Online]. Available: www.bsi.ac.id

J. W. Iskandar and Y. Nataliani, "Perbandingan Naïve Bayes, SVM, dan k-NN untuk Analisis Sentimen Gadget Berbasis Aspek," J. RESTI (Rekayasa Syst. dan Teknol. Inform.), vol. 5, no. 6, pp. 1120–1126, 2021, doi: 10.29207/resti.v5i6.3588.

T. Mardiana, H. Syahreva, and T. Tuslaela, "Komparasi Metode Klasifikasi Pada Analisis Sentimen Usaha Waralaba Berdasarkan Data Twitter," J. Pilar Nusa Mandiri, vol. 15, no. 2, pp. 267–274, Sep. 2019, doi: 10.33480/pilar.v15i2.752.

Salma and W. Silfianti, "Sentiment Analysis of User Review on COVID-19 Information Applications Using Naïve Bayes Classifier, Support Vector Machine, and K-Nearest Neighbors," Int. Res. J. Adv. Eng. Sci., vol. 6, no. 4, pp. 158–162, 2021.

K. S. Maharana, S. Mondal, and B. Nemade, "A review: Data pre-processing and data augmentation techniques," Global Transit. Proc., vol. 3, no. 1, pp. 91–99, 2022.

F. Febriantho, S. Samidi, G. Mikael, and E. Saputra, "Sistem Penentuan Paket Penjualan dengan Algoritma FP-Growth Serta Metode Up dan Cross Selling," J. Media Inform. Budidarma, vol. 6, no. 4, p. 2269, Oct. 2022, doi: 10.30865/mib.v6i4.4800.

C. Schröer, F. Kruse, and J. M. Gómez, "A Systematic Literature Review on Applying CRISP-DM Process Model," in Procedia Comput. Sci., Elsevier B.V., 2021, pp. 526–534, doi: 10.1016/j.procs.2021.01.199.

J. Zheng et al., "Research on complaint prediction based on feature weighted Naive Bayes," in IOP Conf. Ser.: Earth Environ. Sci., vol. 983, no. 1, p. 012117, 2022.

G. Ou, Y. He, P. Fournier-Viger, and J. Z. Huang, "A Novel Mixed-Attribute Fusion-Based Naive Bayesian Classifier," Appl. Sci., vol. 12, no. 20, p. 10443, 2022.

I. Gifari, M. Adha, I. R. Hendrawan, F. Freddy, and S. Durrand, "Analisis Sentimen Review Film Menggunakan TF-IDF dan Support Vector Machine," J. Inf. Technol., vol. 2, no. 1, 2022.

J. Cervantes, F. Garcia-Lamont, L. Rodríguez-Mazahua, and A. Lopez, "A comprehensive survey on support vector machine classification: Applications, challenges and trends," Neurocomputing, vol. 408, pp. 189–215, 2020.

D. M. Abdullah and A. M. Abdulazeez, "Machine learning applications based on SVM classification: A review," Qubahan Acad. J., vol. 1, no. 2, pp. 81–90, 2021.

E. Indrayuni, A. Nurhadi, and D. A. Kristiyanti, “Implementasi Algoritma Naive Bayes, Support Vector Machine, dan K-Nearest Neighbors untuk Analisa Sentimen Aplikasi Halodoc,” Faktor Exacta, vol. 14, no. 2, p. 64, Aug. 2021, doi: 10.30998/faktorexacta.v14i2.9697.

M. Suyal and P. Goyal, "A review on analysis of k-nearest neighbor classification machine learning algorithms based on supervised learning," Int. J. Eng. Trends Technol., vol. 70, no. 7, pp. 43–48, 2022.

D. Cheng, S. Zhang, Z. Deng, Y. Zhu, and M. Zong, "kNN Algorithm with Data-Driven k Value," in Lecture Notes Comput. Sci., Springer, 2014, pp. 499–512, doi: 10.1007/978-3-319-14717-8_39.

Y. Song, X. Kong, and C. Zhang, "A Large‐Scale k‐Nearest Neighbor Classification Algorithm Based on Neighbor Relationship Preservation," Wireless Commun. Mobile Comput., vol. 2022, p. 7409171, 2022.

S. Firman, W. Desena, and A. Wibowo, "Penerapan Algoritma Stemming Nazief & Adriani Pada Proses Klasterisasi Berita Berdasarkan Tematik Pada Laman (Web) Direktorat Jenderal HAM Menggunakan Rapidminer," 2022.