Enhancing AI Feedback Systems: Lessons Learned from a RAG Implementation Study

Abstract

This study investigates the impact of Retrieval-Augmented Generation (RAG) on graduate students' perceptions of AI-generated feedback in a cyber-social learning environment. Using a non-equivalent cohort comparison quasi-experimental design, the research compared survey responses from Fall 2023 (n=96) and Spring 2024 (n=86) before and after implementing RAG. Participants engaged in iterative project cycles involving AI feedback, peer reviews, and instructor evaluations. Results revealed significant improvements in perceived feedback quality (p < .001, d = 0.41) and usefulness (p < .001, d = 0.51) following RAG implementation, while perceptions of actionability remained unchanged (p = .68, d = -0.05). Hierarchical regression analyses identified educational level as a positive predictor of quality and usefulness perceptions, whereas age and location negatively predicted actionability, and educational level positively predicted it. These findings suggest that RAG enhances the contextual relevance and perceived value of AI-generated feedback, particularly for advanced students. However, the lack of improvement in actionability highlights a persistent challenge in AI feedback systems. The study recommends integrating RAG with human support and developing targeted interventions to improve feedback literacy and implementation, especially for less advanced students and older learners.