Toward AI-assisted Exercise Creation for First Course in Programming through Adversarial Examples of AI Models.
Abstract
We propose a new methodology, the Exercise Creation Methodology (ECM), that leverages recent AI technology advancements to create ChatGPT-assisted programming exercises for beginners. ECM takes an existing exercise as input and mutates it by removing some contents into semantically equivalent but syntactically different versions. The pair of versions are labeled as answered correctly and misleadingly by ChatGPT. The removed contents are re-inserted incrementally with further mutation, ensuring the labels remain unchanged. Using the version with the misleading answer and the ChatGPT elaboration on the other version, we construct a ChatGPTassisted exercise. The latter version may also serve as a solution. We illustrate ECM using a case study. Figure 1: ECM Overview solutions. For this purpose, we are developing a novel Exercise Creation Methodology (ECM) as summarized in Section 3 (see also Figure 1). Suppose ChatGPT can produce a correct answer to a programming exercise E , which contains either a textual question to ask students to develop a program or a question with a piece of code for students to further work on specific tasks. For example, suppose E is the sample exercise with code shown in Figure 2 and its list of errors shown in Figure 3 provided to students for them to fix. ChatGPT produces a correct answer shown in Figure 4, which we refer to as f (E). We wonder whether it is feasible to slightly modify E to produce an exercise version M (such as the one in Figure 7b) that ChatGPT will suggest a misleading program f (M ) (Figure 6a), while with a further slightly modified exercise V (Figure 7a), ChatGPT could suggest a correct program f (V ) (Figure 8) similar to the original solution. If this can be done, we can use the exercise version M (that misleads ChatGPT) together with the correct elaboration associated with the answer f (V ) to create an exercise. Moreover, the solution of this ChatGPT-assisted exercise could be f (V ). In this way, we leverage advanced AI models to produce a pair of an exercise and its solution. By the above exercise design, students can attempt an exercise question similar to the past and yet cannot simply use Cha