Pre-trained BERT Architecture Analysis for Indonesian Question Answer Model
DOI:
https://doi.org/10.37385/jaets.v6i1.4746Keywords:
BERT, Chatbot, Indonesian, NLP, Transfer Learning, Question AnswerAbstract
Developing a question-and-answer system in Natural Language Processing (NLP) has become a major concern in the Indonesian language context. One of the main challenges in developing a question-and-answer system is the limited dataset, which can cause instability in system performance. The limitations of the dataset make it difficult for the question-and-answer model to understand and answer questions well. The proposed solution uses Transfer Learning with pre-trained models such as BERT. This research aims to analyze the performance of the BERT model, which has been adapted for question-and-answer tasks in Indonesian. The BERT model uses an Indonesian language dataset adapted specifically for question-and-answer tasks. A customization approach tunes BERT parameters according to the given training data. The results obtained; the model is improved by minimizing the loss function. Evaluation of the trained model shows that the best validation loss is 0.00057 after 150 epochs. In addition, through in-depth evaluation of the similarity of question texts, the BERT model can answer questions measurably, according to existing knowledge in the dataset.
Downloads
References
Alotaibi, R., Ali, A., Alharthi, H., & Almehamdi, R. (2020). AI Chatbot for Tourist Recommendations: A Case Study in the City of Jeddah, Saudi Arabia. International Journal of Interactive Mobile Technologies (iJIM), 14(19), Article 19. https://doi.org/10.3991/ijim.v14i19.17201
Arnumukti, M. L., Sudianto, S., & Athiyah, U. (2023). Product Layout Recommendations based on Customer Behavior and Data Mining. 2023 IEEE International Conference on Communication, Networks and Satellite (COMNETSAT), 330–334. https://doi.org/10.1109/COMNETSAT59769.2023.10420563
Cahyawijaya, S., Winata, G. I., Wilie, B., Vincentio, K., Li, X., Kuncoro, A., Ruder, S., Lim, Z. Y., Bahar, S., Khodra, M. L., Purwarianti, A., & Fung, P. (2021). IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation.
Clark, J. H., Choi, E., Collins, M., Garrette, D., Kwiatkowski, T., Nikolaev, V., & Palomaki, J. (2020). TYDI QA: A Benchmark for Information-Seeking Question Answering in in Typologically Diverse Languages. Transactions of the Association for Computational Linguistics, 8, 454–470. https://doi.org/10.1162/tacl_a_00317
Cortes, E. G., Woloszyn, V., Barone, D., Möller, S., & Vieira, R. (2022). A systematic review of question answering systems for non-factoid questions. Journal of Intelligent Information Systems, 58(3), 453–480. https://doi.org/10.1007/s10844-021-00655-8
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (arXiv:1810.04805). arXiv. https://doi.org/10.48550/arXiv.1810.04805
Dikti, K. M. (2020). Buku Panduan Merdeka Belajar—Kampus Merdeka.
Galassi, A., Lippi, M., & Torroni, P. (2021). Attention in Natural Language Processing. IEEE Transactions on Neural Networks and Learning Systems, 32(10), 4291–4308. https://doi.org/10.1109/TNNLS.2020.3019893
Imamura, K., & Sumita, E. (2019). Recycling a Pre-trained BERT Encoder for Neural Machine Translation. In A. Birch, A. Finch, H. Hayashi, I. Konstas, T. Luong, G. Neubig, Y. Oda, & K. Sudoh (Eds.), Proceedings of the 3rd Workshop on Neural Generation and Translation (pp. 23–31). Association for Computational Linguistics. https://doi.org/10.18653/v1/D19-5603
Indah Rahajeng, M., & Purwarianti, A. (2021). Indonesian Question Answering System for Factoid Questions using Face Beauty Products Knowledge Graph. Jurnal Linguistik Komputasional (JLK), 4(2), 59. https://doi.org/10.26418/jlk.v4i2.62
Jhaerol, M. R., & Sudianto, S. (2023). Implementation of Chatbot for Merdeka Belajar Kampus Merdeka Program using Long Short-Term Memory. Jurnal Nasional Pendidikan Teknik Informatika?: JANAPATI, 12(2), Article 2. https://doi.org/10.23887/janapati.v12i2.58794
Koto, F., Rahimi, A., Lau, J. H., & Baldwin, T. (2020). IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP. Proceedings of the 28th International Conference on Computational Linguistics, 757–770. https://doi.org/10.18653/v1/2020.coling-main.66
Kovaleva, O., Romanov, A., Rogers, A., & Rumshisky, A. (2019). Revealing the Dark Secrets of BERT (arXiv:1908.08593). arXiv. http://arxiv.org/abs/1908.08593
Lasama, J., Sudianto, S., Ramadhani, R., Hilmawan, M. D., Aldean, M. Y., & Satria, M. A. H. (2024). English Indonesia-Chan: OPUS-MT Powered Chatbot. Jurnal Teknik Elektro dan Komputasi (ELKOM), 6(1), 105–111. https://doi.org/10.32528/elkom.v6i1.18613
Lee, J.-S., & Hsiang, J. (2020). Patent classification by fine-tuning BERT language model. World Patent Information, 61, 101965. https://doi.org/10.1016/j.wpi.2020.101965
Liu, P. J., Saleh, M., Pot, E., Goodrich, B., Sepassi, R., Kaiser, L., & Shazeer, N. (2018). Generating Wikipedia by Summarizing Long Sequences (arXiv:1801.10198). arXiv. https://doi.org/10.48550/arXiv.1801.10198
Lubis, A., & Sumartono, I. (2023). Implementasi Layanan Akademik Berbasis Chatbot untuk Meningkatkan Interaksi Mahasiswa. 3(5).
Nasa-Ngium, P., Nuankaew, W. S., & Nuankaew, P. (2023). Analyzing and Tracking Student Educational Program Interests on Social Media with Chatbots Platform and Text Analytics. International Journal of Interactive Mobile Technologies (iJIM), 17(05), Article 05. https://doi.org/10.3991/ijim.v17i05.31593
Naufal, A. B., Sudianto, S., & Fachri, M. A. A. (2023). Implementation of Chatbot System on Tourism Objects in Banyumas Regency with AIML and Chatterbot. 5.
Nguyen, T. T., Le, A. D., Hoang, H. T., & Nguyen, T. (2021). NEU-chatbot: Chatbot for admission of National Economics University. Computers and Education: Artificial Intelligence, 2. https://doi.org/10.1016/j.caeai.2021.100036
Pratama, T., & Rjito, S. (2021). IndoXLNet: Pre-Trained Language Model for Bahasa Indonesia. International Journal of Engineering Trends and Technology, 70(5), 367–381. https://doi.org/10.14445/22315381/IJETT-V70I5P240
Schwartz, R., Dodge, J., Smith, N. A., & Etzioni, O. (2020). Green AI. Communications of the ACM, 63(12), 54–63. https://doi.org/10.1145/3381831
Sudianto, Herdiyeni, Y., & Prasetyo, L. B. (2023a). Machine learning for sugarcane mapping based on segmentation in cloud platform. AIP Conference Proceedings, 2482(1), 020001. https://doi.org/10.1063/5.0132180
Sudianto, S., Herdiyeni, Y., & Prasetyo, L. B. (2023b). Classification of Sugarcane Area Using Landsat 8 and Random Forest based on Phenology Knowledge. JOIV?: International Journal on Informatics Visualization, 7(3–2), Article 3–2. https://doi.org/10.30630/joiv.7.3-2.1401
Sudianto, S., Herdiyeni, Y., & Prasetyo, L. B. (2023c). Early Warning for Sugarcane Growth using Phenology-Based Remote Sensing by Region. International Journal of Advanced Computer Science and Applications, 14(2). https://doi.org/10.14569/IJACSA.2023.0140259
Wilie, B., Vincentio, K., Winata, G. I., Cahyawijaya, S., Li, X., Lim, Z. Y., Soleman, S., Mahendra, R., Fung, P., Bahar, S., & Purwarianti, A. (2020). IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding.