Adoption of Computer Programming Exercises for Automatic Assessment - Issues and Caution.
Abstract
Computational thinking is an interdisciplinary core skill to be acquired in STEM education, while computer program coding is a concrete manifestation of such a skill. In response to the increasing size of computer programming classes and rapidly growing number of learners, particularly in massive open online courses (MOOCs), many instructors nowadays heavily rely on the use of automated systems to assess the programming work of students. However, these automated assessment systems typically perform black box testing to determine the correctness of student programs, which limits the type of programming exercises that can be automatically assessed. This paper reports a case study on the adoption of programming exercises from textbook and online resources, and categorises some difficulties and issues of caution due to the technical limitation of typical automated assessment systems. The identified issues are mainly related to the input/output and non-deterministic nature of the programs or the intended learning outcomes of some of the exercises. The paper concludes with a brief outline of recent research directions to alleviate these problems for improvement of learning. Keywords: Assessment of learning, automated assessment, black box testing, computational thinking, computer programming exercises, technology-enhanced learning and assessment 1. Introduction Computational thinking is an interdisciplinary core skill to be acquired in STEM education with relevance across all four disciplines of science, technology, engineering and mathematics, while computer programming (or program coding) is a concrete manifestation of such a skill. Computer programming is now taught not only at tertiary and senior secondary levels of education, but increasingly at junior high schools or even primary schools. It is also a core subject of almost all STEM-related majors spanning across different faculties and schools in most universities, and is now a popular subject in general education for students from non-STEM majors. It is not unusual to have hundreds of students attending a computer programming course at the same time (Wang & Wong, 2007), particularly in massive open online cou