Identifying the cause of performance issues of Pretrained Language Model for Educational Technology
Abstract
Online self-learning platforms, such as questionanswering (Q&A) websites, are popular technology for learning, and utilising Pretrained Language Models (PLMs) to maintain their content qualities is a common practice. However, it is challenging to identify the cause that affects the performance of PLMs. In this study, we propose using machine common sense to identify the cause affecting the performance of PLMs. We conducted an empirical experiment with three PLMs using a publicly available dataset. We select 45000 data points as the training data and 1000 as the testing data. We first train the PLMs with training data, then run machine common sense tests to examine their reasoning abilities. We define the causal relationship between content quality and reasoning ability, and use the cause to derive Metamorphic Relations (MR) of Metamorphic Testing (MT) for creating follow-up testing datasets. We analyse the changes between source and follow-up outputs to see whether the identified cause affects the performance. Results show that the reasonableness of the content is the cause that affects the performance of PLM, which has reasoning abilities. In addition, the proposed approach in this study is effective for identifying and validating the cause that affects the performance of PLMs, even on devices with limited computer resources. Future research can apply our approach and seek different machine common sense tests and counterfactual analysing techniques to identify different causes of performance issues of different PLMs. approach is effective on distilled version BERT-based PLM and that improving the reasonableness of content can improve this PLM's performance in content quality classification. To the best of our knowledge, we are the first to introduce this novel approach for identifying and validating the cause that affects the performance of PLM-based educational technology. The remainder of this paper is structured as follows: Section II reviews on the self-learning platform with PLM, causality, machine common sense, and MT. Section III presents an overview of the evaluation methodology, including the proposed MRs, experiment design, and implementation proce