Skip to main content

Large language model generation of plausible semantic distractors for data science Parsons problems

PDF

Abstract

Parsons problems have been widely recognised to support novice learners by reducing the cognitive demands of writing code from scratch through rearrangement of provided code blocks. A common variation includes distractors – erroneous blocks of code which do not belong to the correct solution. While Parsons problems have been widely studied for introductory programming, they remain underexplored in various domains including in data science education, where students may possess statistical knowledge but limited programming experience. Developing distractors is also time-consuming for educators. This study investigated the feasibility of using a large language model (LLM) to generate semantic distractors for data science Parsons problems. A set of distractors was generated for 315 Python-based data science Parsons problems and was evaluated for pedagogical plausibility, compilability, and output divergence. Validated distractors were then incorporated as exemplars in a refined prompt to generate a second set of distractors, which were evaluated using the same criteria. The findings demonstrate that LLMs can be used to develop semantic distractors for data science Parsons problems and refining the prompts using validated exemplars further improved the quality. This study contributes to the limited literature on data science Parsons problems, distractor design and LLM-supported development of these problems.

Keywords

Parsons problems, distractors, Data Science, Large Language Models (LLMs), quantitative.