Abstract
Generative AI has not changed the need for assessment, but it has changed what certain forms of assessment can reliably tell us about student performance. When students can access AI support for idea generation and technical implementation, what assessment should measure becomes an open question (Perkins et al., 2024) and aggregate course grades may provide little insight to distinguish strong conceptual work from competent production. We report on a preliminary analysis of rubric data from a postgraduate data visualisation course (n =175) where AI use for idea generation and coding support was explicitly permitted and treated as a normal part of the workflow. The course assessment was redesigned around capability-level rubrics to capture what matters most when technical production can be AI-assisted.
The assessment design was informed by evaluative judgement theory (Bearman et al., 2024; Tai et al., 2018), where the capacity to recognise and justify quality is positioned as a graduate capability developed through repeated engagement with standards. The course used descriptive capability rubrics with five levels from Foundation to Exceptional, applied across multiple assessments and a final project portfolio. As Gyamfi et al. (2022) demonstrate, descriptive levels preserve the interpretive work that evaluative judgement requires rather than orienting students toward compliance. A separate competency review assessed critique, interpretation, and application of visualisation principles under exam conditions without AI access.
The final grade distribution looked similar to previous iterations of the course, but the capability-level rubric data had a more interesting story to tell. Distinct student profiles emerged, including those whose technical implementation capability exceeded their ability to communicate a coherent data story, suggesting a gap between procedural and deeper conceptual capability (Biggs & Tang, 2011). A similar pattern was visible in a dedicated storytelling assessment, where strong visuals were not consistently matched by strong narrative performance. These within-student differences were largely invisible in aggregate grades. The competency review provided an independent measure of evaluative judgement: what students could do without AI assistance or visualisation making tools. Competency review performance showed the strongest though modest association with the portfolio data story criterion, suggesting that the ability to interpret and communicate was the more reliable indicator of independent judgement (Bearman et al., 2024).
We argue that rubric data, already generated through normal marking processes in most LMS environments, constitutes an under-utilised source of learning analytics. Examined at the level of capability profiles rather than aggregate scores, this approach can surface distinctions between production and judgement that grades conceal across many discipline contexts, providing actionable insights that connect assessment evidence to teaching decisions (Kennedy et al., 2025). Rather than framing AI as a threat to assessment integrity, we ask a different question: if technical production is increasingly supported, which capabilities remain the most trustworthy evidence of what students know? As argued by Corbin et al. (2026), deciding which forms of evidence deserve the greatest trust may matter more than detecting whether AI was used. We show that capability-based rubric data may be one such form of trustworthy evidence.
Keywords
Capability-based assessment, evaluative judgement, generative AI, learning analytics.
