Comparative Analysis of Data-Level and Cost-Sensitive Learning in IndoBERT-Based Sentiment Analysis of Ruangguru App Reviews

Ade Toti Febrian, Purwadi Purwadi, Adam Prayogo Kuncoro

Abstract


User reviews of online learning applications such as Ruangguru provide valuable information for evaluating service quality, user experience, and digital learning effectiveness. Although IndoBERT has demonstrated strong performance in Indonesian sentiment analysis, previous studies generally compared imbalance handling techniques using different datasets, model architectures, and experimental protocols, making the relative effectiveness of data-level and cost-sensitive learning approaches difficult to evaluate objectively under the same Transformer backbone. This study compares a Data-Level Approach using Latent-SMOTE with a Cost-Sensitive Learning Approach using Class-Weighted Loss on an identical IndoBERT architecture. The dataset consists of 3,767 Ruangguru user reviews collected from Google Play Store and processed through text preprocessing, IndoBERT tokenization, stratified train-validation-test splitting, and evaluation using Accuracy, Precision, Recall, Macro F1-score, confusion matrix, Cochran's Q Test, and McNemar Test. Experimental results show that the Baseline model achieved the highest Accuracy (90.05%), while the Cost-Sensitive Learning approach obtained the highest Macro F1-score (0.6275), outperforming both the Baseline (0.5658) and the Data-Level approach. These findings indicate that class-weighted optimization improves minority-class recognition without modifying the original training distribution, whereas Latent-SMOTE enhances minority representation but does not outperform Class-Weighted Loss. McNemar testing further confirms that the improvements over the Baseline are statistically significant. The main contribution of this work is an objective comparison of Data-Level and Cost-Sensitive Learning approaches using the same IndoBERT backbone, dataset, preprocessing pipeline, hyperparameters, and evaluation protocol. In addition, the study applies Latent-SMOTE in the latent feature space and complements performance evaluation with statistical significance testing, providing stronger empirical evidence for handling imbalanced Indonesian sentiment datasets


Keywords


Sentiment Analysis; IndoBERT; Transformer; Natural Language Processing; Class Imbalance; Imbalanced Learning; Data-Level Approach; Cost-Sensitive Learning Approach; Mobile Application Reviews.

Full Text:

PDF

References


A. R. Putra and D. E. Ratnawati, "Analisis Sentimen Berbasis Aspek pada Aplikasi Mobile Menggunakan Naïve Bayes Berdasarkan Ulasan Pengguna Google Play Store (Studi Kasus: JConnect Mobile)," Jurnal Teknologi Informasi dan Ilmu Komputer, vol. 12, no. 2, pp. 293–300, 2025, doi: 10.25126/jtiik.2025127556.

I. S. K. Idris, Y. A. Mustofa, and I. A. Salihi, "Analisis Sentimen terhadap Penggunaan Aplikasi Shopee Menggunakan Algoritma Support Vector Machine (SVM)," Jambura Journal of Electrical and Electronics Engineering, vol. 5, no. 1, pp. 32–35, 2023, doi: 10.37905/jjeee.v5i1.16830.

Y. A. Mustofa and I. S. K. Idris, "Ensemble Approach to Sentiment Analysis of Google Play Store App Reviews," Jambura Journal of Electrical and Electronics Engineering, vol. 6, no. 2, pp. 181–188, 2024, doi: 10.37905/jjeee.v6i2.25184.

A. Hadiyan, M. Rafli, F. Nugroho, and M. F. Luthfia, "Analisis Sentimen Komentar Pengguna terhadap Aplikasi Prime Video di Google Play Store dengan Pendekatan Machine Learning," Jurnal Teknologi Informasi dan Komputer, vol. 6, no. 4, pp. 166–173, 2025.

D. Tribuana, U. Usman, and D. Dayanti, "Penerapan Natural Language Processing untuk Analisis Sentimen terhadap Layanan Publik di Media Sosial Twitter," Jurnal Teknologi dan Bisnis Cerdas, vol. 1, no. 1, pp. 28–37, 2025, doi: 10.64476/jtbc.v1i1.3.

G. Fani and S. Medantoro, "Comparative Analysis of IndoBERT and Classic Machine Learning Models for Sentiment Classification of Education Policy on Social Media X," Journal of Computer Science and Software, vol. 10, no. 1, pp. 548–557, 2026.

T. Indriyatmoko, M. Rahardi, H. Utama, A. C. Frobenius, and B. Multilanguage, "Comparative Analysis of BERT and LSTM Models for Sentiment Classification of Mobile Game User Reviews," Journal of Information Technology, vol. 10, no. 1, pp. 1093–1100, 2026.

A. N. Wahyuni et al., "Implementasi Model IndoBERT dan mBERT untuk Klasifikasi Teks Berbahasa Indonesia," Jurnal Teknologi Komputer, vol. 10, no. 2, pp. 2736–2743, 2026.

J. R. Sari, K. Sadik, A. M. Soleh, and C. Suhaeni, "Evaluasi Model Klasifikasi dalam Deteksi Penipuan Transaksi: Studi Kasus pada Data Tidak Seimbang," Jurnal Gaussian, vol. 14, no. 2, pp. 565–576, 2025, doi: 10.14710/j.gauss.14.2.565-576.

A. Karim, B. Bangun, S. Prayetno, and M. Afrendi, "Optimasi Prediksi Harga Sawit Menggunakan Teknik Stacking Algoritma Machine Learning dan Deep Learning dengan SMOTE," Building Informatics, Technology and Science (BITS), vol. 7, no. 1, pp. 638–645, 2025, doi: 10.47065/bits.v7i1.7239..

B. Pes and G. Lai, "Cost-Sensitive Learning Strategies for High-Dimensional and Imbalanced Data: A Comparative Study," PeerJ Computer Science, vol. 7, p. e832, 2021, doi: 10.7717/peerj-cs.832.

H. Hairani, T. Widiyaningtyas, and D. D. Prasetya, "Addressing Class Imbalance of Health Data: A Systematic Literature Review on Modified Synthetic Minority Oversampling Technique (SMOTE) Strategies," International Journal of Informatics and Visualization, vol. 8, no. 3, pp. 1310–1318, 2024, doi: 10.62527/joiv.8.3.2283.

N. Isma, L. T. Syahyaningsih, D. F. Surianto, and N. Fadilah, "Analysis of Opinion Classification on Marriage Based on Support Vector Machine and Multi-Layer Perceptron," Jambura Journal of Electrical and Electronics Engineering, vol. 8, no. 1, pp. 1–7, 2026, doi: 10.37905/jjeee.v8i1.29616.

A. B. S. Br. Sembiring, R. Robet, and L. Hoki, "Comparison of IndoBERT and SVM Performance in Sentiment Analysis of Digital Education Platforms," Sinkron, vol. 10, no. 1, pp. 64–74, 2026, doi: 10.33395/sinkron.v10i1.15472.

R. Anadra, H. Wijayanto, and K. Sadik, "Sentiment Analysis of Tokopedia Customer Reviews Using BiLSTM and IndoBERT with Comparative Analysis of Preprocessing and Labeling Methods," International Journal of Advances in Data and Information Systems, vol. 6, no. 3, pp. 773–788, 2025, doi: 10.59395/ijadis.v6i3.1458.

R. J. Siger, M. Naufal, and F. Alzami, "Analisis Performa Class Weight dan Focal Loss pada Model IndoBERT untuk Klasifikasi Teks Depresi Berbahasa Indonesia," JURIKOM (Jurnal Riset Komputer), vol. 13, no. 2, pp. 558–568, 2026, doi: 10.30865/jurikom.v13i2.9620.

L. D. Cahya, A. Luthfiarta, J. I. T. Krisna, S. Winarno, and A. Nugraha, "Improving Multi-label Classification Performance on Imbalanced Datasets Through SMOTE Technique and Data Augmentation Using IndoBERT Model," Jurnal Nasional Teknologi dan Sistem Informasi, vol. 9, no. 3, pp. 290–298, 2024, doi: 10.25077/teknosi.v9i3.2023.290-298.

F. Y. A'la, "Optimasi Klasifikasi Sentimen Ulasan Game Berbahasa Indonesia: IndoBERT dan SMOTE untuk Menangani Ketidakseimbangan Kelas," Edumatic: Jurnal Pendidikan Informatika, vol. 9, no. 1, pp. 256–265, 2025, doi: 10.29408/edumatic.v9i1.29666.

I. Saputra and G. L. Ginting, "Sentiment Analysis of Tokopedia Customer Reviews Using IndoBERT and SMOTE for Class Imbalance Handling," Journal of Online Information Systems and Computing, vol. 7, no. 1, pp. 20–27, 2025, doi: 10.47065/josyc.v7i1.8748.

M. R. Roihan, M. I. Mazdadi, I. Budiman, and F. Indriani, "Comparative Performance of IndoBERT-Based Deep Learning Models with SMOTE for Trade Tariff Sentiment Analysis," Journal of Big Data Research, pp. 432–446, 2026..

R. Widayanti and F. C. Kasih, "Comparative Analysis of Baseline IndoBERT, Class-Weighted IndoBERT, and SMOTE with Support Vector Machine for Handling Imbalanced Sentiment Classification in Indonesian," Jurnal Teknik Informatika (JUTIF), vol. 7, no. 3, 2026.

A. S. Agil Rafsanjani, D. L. Fithri, and S. Supriyono, "Sentiment Analysis of User Reviews of the KitaLulus Application on Google Play Store Using the Support Vector Machine (SVM) Algorithm," Sistemasi, vol. 14, no. 5, 2025, doi: 10.32520/stmsi.v14i5.5519.

B. Siswanto and M. Hanafi, "Optimasi IndoBERT untuk Pengenalan Entitas Bernama Bahasa Indonesia pada Data Media Sosial dengan Penalaan Hiperparameter Optuna," JURIKOM (Jurnal Riset Komputer), vol. 13, no. 1, pp. 221–233, 2026, doi: 10.30865/jurikom.v13i1.9545.

R. I. Perwira, V. A. Permadi, D. I. Purnamasari, and R. P. Agusdin, "Domain-Specific Fine-Tuning of IndoBERT for Aspect-Based Sentiment Analysis in Indonesian Travel User-Generated Content," Journal of Information Systems Engineering and Business Intelligence, vol. 11, no. 1, pp. 30–40, 2025, doi: 10.20473/jisebi.11.1.30-40.

I. Mursidah et al., "Klasifikasi Sentimen Google Play Store Aplikasi ChatGPT Berbahasa Indonesia Berbasis IndoBERT," Jurnal Minfo Polgan, vol. 14, no. 2, pp. 3349–3359, 2025, doi: 10.33395/jmp.v14i2.15751.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," in Proc. 2019 Conf. North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), Minneapolis, MN, USA, 2019, pp. 4171–4186.

A. R. Illawati, A. I. Hadiana, and M. Melina, "Klasifikasi Penyakit Monkeypox dengan XGBoost dan SMOTE untuk Penanganan Data Tidak Seimbang," Jurnal Algoritma, vol. 22, no. 2, pp. 35–44, 2025, doi: 10.33364/algoritma/v.22-2.2349.

R. Ishak and A. Bengnga, "Optimizing IndoBERT Embedding with Ditto Whitening for Measuring Research Title Similarity," Jambura Journal of Electrical and Electronics Engineering, vol. 8, no. 1, pp. 46–54, 2026, doi: 10.37905/jjeee.v8i1.35554.

W. Clarisha, A. A. M. Fani, D. F. Surianto, and N. Fadilah, "Sentiment Analysis of Local Sunscreen Skintific, Somethinc, and Avoskin with Naïve Bayes and SVM," Jambura Journal of Electrical and Electronics Engineering, vol. 7, no. 2, pp. 264–271, 2025, doi: 10.37905/jjeee.v7i2.30257.

K. M. Sujon, R. Hassan, K. Choi, and M. A. Samad, "Accuracy, Precision, Recall, F1-Score, or MCC? Empirical Evidence from Advanced Statistics, Machine Learning, and Explainable AI for Evaluating Business Predictive Models," Journal of Big Data, vol. 12, no. 1, 2025, doi: 10.1186/s40537-025-01313-4..

A. C. Ghazy, G. Ghozali, and K. A. Wibowo, "Transformasi Pendidikan: Pengembangan Metodologi dan Media Pembelajaran di Era Digital," Action Research Journal Indonesia, vol. 7, no. 4, 2025, doi: 10.61227/arji.v7i4.594.

M. A. Aljauhari and F. Arifin, "Implementasi SMOTE pada Klasifikasi Email Spam Menggunakan Algoritma Machine Learning Berbasis TF-IDF," Jurnal Algoritma, vol. 23, no. 1, 2026, doi: 10.33364/algoritma.v23i1.3246.

A. N. Laily, M. A. Barata, and D. Nurdiansyah, "Analisis Klasifikasi Hepatitis Menggunakan Synthetic Minority Oversampling Technique, Support Vector Machine, dan Random Forest," Decode: Jurnal Pendidikan Teknologi Informasi, vol. 6, no. 1, pp. 223–236, 2026, doi: 10.51454/decode.v6i1.1630.

A. N. Putro and M. A. Muslim, "Penanganan Ketidakseimbangan Data Ekstrim pada Sistem Prediksi," Techno.Com, vol. 24, no. 4, pp. 1359–1367, 2025, doi: 10.62411/tc.v24i4.15005.

M. A. A. Saputra et al., "IndoBERT-Relevancy: A Context-Conditioned Relevancy Classifier for Indonesian Text," arXiv preprint arXiv:2603.26095, pp. 1–9, 2026.

A. Z. Muhabbab, B. Bunyamin, and H. Hasmawati, "Sentiment Analysis of Indonesia's Free Nutritious Meal Program on Platform X (Formerly Twitter) Using IndoBERT," ZERO: Jurnal Sains, Matematika, dan Terapan, vol. 9, no. 3, 2025, doi: 10.30829/zero.v9i3.27629.

B. E. S. Dewi et al., "Sentimen Kendaraan Listrik Menggunakan Fine-Tuning IndoBERT dan IndoBERTweet," Antivirus Journal, vol. 19, no. 1, pp. 45–56, 2025.

W. Widyananda et al., "Machine Learning and Transformer-Based Model for Indonesian E-Commerce Review Sentiment Analysis," International Journal of Computer Science, vol. 14, no. 2, pp. 112–125, 2025.

Akbar et al., "Klasifikasi Sentimen untuk Mengetahui Kecenderungan Opini Politik pada Media Sosial X Berbasis IndoBERT," Building Informatics, Technology and Science (BITS), vol. 6, no. 3, pp. 1420–1429, 2024.

M. B. Nugroho et al., "Multiclass Sentiment Analysis of Electric Vehicle Incentive Policy Using IndoBERT and DeBERTa," Journal of Applied Informatics and Computing (JAIC), vol. 9, no. 1, pp. 88–97, 2025.




DOI: https://doi.org/10.37905/jjeee.v8i2.39885

Refbacks

  • There are currently no refbacks.


Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Published by:
Electrical Engineering Department
Faculty of Engineering
State University of Gorontalo
Jalan B.J.Habibie Desa Moutong Kecamatan Tilongkabila Kabupaten Bone Bolango
Telp. 0435-821175; 081340032063
Email: [email protected]/[email protected]

Creative Commons License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.