Artificial intelligence in higher education learning: transferable skills and academic integrity
DOI:
https://doi.org/10.70770/vrz6qe26Keywords:
generative AI, Pedagogical practices, learning and teaching, higher education, engineeringAbstract
The advancement of Generative Artificial Intelligence (AI) chatbots, such as ChatGPT, presents significant and transformative challenges in higher education teaching and learning, such as assessment and evaluation practices. While this is acknowledged, there has been very little research into what this might look like in daily practice in higher education. This study explored these challenges in one area of higher education practice: developing students’ transferable skills, including writing, critical thinking, and information literacy among undergraduate engineering students at RMIT University, Melbourne, Australia. Using a cohort comparison design, this study evaluated the impact of ChatGPT on students' attainment of transferable skills. The effectiveness of AI tools in enhancing educational outcomes was assessed with a standardised assessment framework used by independent assessors to grade students’ reports. The results, analysed using the Mann-Whitney U test and the intraclass correlation coefficient, revealed significant improvements in critical thinking and information literacy among those students who used ChatGPT. The study also explored the ethical implications of using AI in educational settings and highlighted the need for rigorous academic standards and the implementation of measures to ensure the responsible use of AI technologies. While the preliminary findings suggest that AI tools, particularly ChatGPT in this study, can positively impact certain students’ skills, more detailed and controlled studies are necessary to validate these results and explore further the mechanisms through which AI tools influence learning and skill development.
Downloads
References
Alavi, M., Biros, E., & Cleary, M. (2022). A primer of inter-rater reliability in clinical measurement studies: Pros and pitfalls. Journal of Clinical Nursing, 31(23-24), e39-e42. https://doi.org/10.1111/jocn.16514
Alin, P., Arendt, A., & Gurell, S. (2023). Addressing cheating in virtual proctored examinations: toward a framework of relevant mitigation strategies. Assessment & Evaluation in Higher Education, 48(3), 262-275. https://doi.org/10.1080/02602938.2022.2075317
Anastasia, O., Saini, A., Zapata, G., Searsmith, D., Cope, B., Kalantzis, M., Castro, V., Kourkoulou, T., Jones, J., da Silva, R. A., Whiting, J., & Nikoleta Polyxeni, K. (2023). Generative AI: Implications and Applications for Education. arXiv preprint. https://doi.org/10.48550/arXiv.2305.07605
Baidoo-Anu, D., Asamoah, D., Amoako, I., & Mahama, I. (2024). Exploring student perspectives on generative artificial intelligence in higher education learning. Discover Education, 3(1), 98. https://doi.org/10.1007/s44217-024-00173-z
Barrett, A., & Pack, A. (2023). Not quite eye to A.I.: student and teacher perspectives on the use of generative artificial intelligence in the writing process. International Journal of Educational Technology in Higher Education, 20(1), 59. https://doi.org/10.1186/s41239-023-00427-0
Batista, J., Mesquita, A., & Carnaz, G. (2024). Generative AI and Higher Education: Trends, Challenges, and Future Directions from a Systematic Literature Review. Information (Basel), 15(11), 676. https://doi.org/10.3390/info15110676
Burns, A., de Bruyn, L., & Wilson, S. (2022). A rubric approach to assessing information literacy competency in tertiary curricula. Journal of university teaching & learning practice, 20(1). https://doi.org/10.53761/1.20.01.10
Buruk, O. O. (2023). Academic Writing with GPT-3.5: Reflections on Practices, Efficacy and Transparency. https://doi.org/10.48550/arxiv.2304.11079
Chan, C. K. Y. (2023). A comprehensive AI policy education framework for university teaching and learning. International Journal of Educational Technology in Higher Education, 20(1), 38. https://doi.org/10.1186/s41239-023-00408-3
Chan, C. K. Y., & Hu, W. (2023). Students’ voices on generative AI: perceptions, benefits, and challenges in higher education. International Journal of Educational Technology in Higher Education, 20(1), 43. https://doi.org/10.1186/s41239-023-00411-8
Cooper, G. (2023). Examining Science Education in ChatGPT: An Exploratory Study of Generative Artificial Intelligence. Journal of Science Education and Technology, 32(3), 444-452. https://doi.org/10.1007/s10956-023-10039-y
Cope, B., & Kalantzis, M. (2023). Platformed Learning: Reshaping Education in the Era of Learning Management Systems. Varieties of Platformisation: Critical Perspectives on EdTech in Higher Education. London: Palgrave Macmillan.
Cotton, D. R. E., Cotton, P. A., & Shipway, J. R. (2023). Chatting and cheating: Ensuring academic integrity in the era of ChatGPT. Innovations in Education and Teaching International, 1-12. https://doi.org/10.1080/14703297.2023.2190148
Dai, W., Lin, J., Jin, F., Li, T., Tsai, Y.-S., Gašević, D., & Chen, G. (2023). Can Large Language Models Provide Feedback to Students? A Case Study on ChatGPT. 2023 IEEE International Conference on Advanced Learning Technologies (ICALT), 323-325. https://doi.org/10.1109/ICALT58122.2023.00100
Dai, Y., Liu, A., & Lim, C. P. (2023). Reconceptualizing ChatGPT and generative AI as a student-driven innovation in higher education. Procedia CIRP, 119, 84-90. https://doi.org/10.1016/j.procir.2023.05.002
Darvishi, A., Khosravi, H., Sadiq, S., Gašević, D., & Siemens, G. (2024). Impact of AI assistance on student agency. Computers & Education, 210, 104967. https://doi.org/10.1016/j.compedu.2023.104967
Ellikkal, A., & Rajamohan, S. (2024). AI-enabled personalized learning: empowering management students for improving engagement and academic performance. Vilakshan - XIMB Journal of Management, ahead-of-print(ahead-of-print). https://doi.org/10.1108/XJM-02-2024-0023
Essien, A., Bukoye, O. T., O’Dea, X., & Kremantzis, M. (2024). The influence of AI text generators on critical thinking skills in UK business schools. Studies in Higher Education, 49(5), 865-882. https://doi.org/10.1080/03075079.2024.2316881
Farazouli, A., Cerratto-Pargman, T., Bolander-Laksov, K., & McGrath, C. (2023). Hello GPT! Goodbye home examination? An exploratory study of AI chatbots impact on university teachers’ assessment practices. Assessment & Evaluation in Higher Education, 1-13. https://doi.org/10.1080/02602938.2023.2241676
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023). How Does ChatGPT Perform on the United States Medical Licensing Examination? The Implications of Large Language Models for Medical Education and Knowledge Assessment. JMIR Med Educ, 9, e45312. https://doi.org/10.2196/45312
Harris, D., Coleman, K., & Cook, P. J. (2022). Radical rubrics: implementing the critical and creative thinking general capability through an ecological approach. The Australian Educational Researcher. https://doi.org/10.1007/s13384-022-00521-8
Huq, A., & Gilbert, D. (2017). All the world’s a stage: transforming entrepreneurship education through design thinking. Education & training (London), 59(2), 155-170. https://doi.org/10.1108/ET-12-2015-0111
Johnston, H., Wells, R. F., Shanks, E. M., Boey, T., & Parsons, B. N. (2024). Student perspectives on the use of generative artificial intelligence technologies in higher education. International Journal for Educational Integrity, 20(1), 2. https://doi.org/10.1007/s40979-024-00149-4
Kamalov, F., Santandreu Calonge, D., & Gurrib, I. (2023). New Era of Artificial Intelligence in Education: Towards a Sustainable Multifaceted Revolution. Sustainability, 15(16), 12451. https://www.mdpi.com/2071-1050/15/16/12451
Kaplan-Rakowski, R., Grotewold, K., Hartwick, P., & Papin, K. (2023). Generative AI and Teachers' Perspectives on Its Implementation in Education. Journal of Interactive Learning Research, 34, 313-338. https://www.learntechlib.org/primary/p/222363/
Khan, R. A., Jawaid, M., Khan, A. R., & Sajjad, M. (2023). ChatGPT-Reshaping medical education and clinical management. Pakistan journal of medical sciences, 39(2), 605-607. https://doi.org/10.12669/pjms.39.2.7653
Khlaif, Z. N., Ayyoub, A., Hamamra, B., Bensalem, E., Mitwally, M. A. A., Ayyoub, A., Hattab, M. K., & Shadid, F. (2024). University Teachers’ Views on the Adoption and Integration of Generative AI Tools for Student Assessment in Higher Education. Education Sciences, 14(10), 1090. https://www.mdpi.com/2227-7102/14/10/1090
Kim, J., Yu, S., Detrick, R., & Li, N. (2024). Exploring students’ perspectives on Generative AI-assisted academic writing. Education and Information Technologies. https://doi.org/10.1007/s10639-024-12878-7
Kizilcec, R. F., Huber, E., Papanastasiou, E. C., Cram, A., Makridis, C. A., Smolansky, A., Zeivots, S., & Raduescu, C. (2024). Perceived impact of generative AI on assessments: Comparing educator and student perspectives in Australia, Cyprus, and the United States. Computers and Education: Artificial Intelligence, 7, 100269. https://doi.org/https://doi.org/10.1016/j.caeai.2024.100269
Koo, T. K., & Li, M. Y. (2016). A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. Journal of Chiropractic Medicine, 15(2), 155-163. https://doi.org/10.1016/j.jcm.2016.02.012
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., & Tseng, V. (2023). Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digital Health, 2(2), e0000198. https://doi.org/10.1371/journal.pdig.0000198
Kurtz, G., Amzalag, M., Shaked, N., Zaguri, Y., Kohen-Vacs, D., Gal, E., Zailer, G., & Barak-Medina, E. (2024). Strategies for Integrating Generative AI into Higher Education: Navigating Challenges and Leveraging Opportunities. Education Sciences, 14(5), 503. https://www.mdpi.com/2227-7102/14/5/503
Lane, H. C., McCalla, G., Looi, C.-K., & Bull, S. (2016). Preface to the IJAIED 25th Anniversary Issue, Part 2. International Journal of Artificial Intelligence in Education, 26(2), 539-543. https://doi.org/10.1007/s40593-016-0109-9
Lee, D., Arnold, M., Srivastava, A., Plastow, K., Strelan, P., Ploeckl, F., Lekkas, D., & Palmer, E. (2024). The impact of generative AI on higher education learning and teaching: A study of educators’ perspectives. Computers and Education: Artificial Intelligence, 6, 100221. https://doi.org/10.1016/j.caeai.2024.100221
Li, Z., & Xu, Y. (2023). Designing a realistic peer-like embodied conversational agent for supporting children storytelling. https://doi.org/10.48550/arxiv.2304.09399
Lim, W. M., Gunasekara, A., Pallant, J. L., Pallant, J. I., & Pechenkina, E. (2023). Generative AI and the future of education: Ragnarök or reformation? A paradoxical perspective from management educators. The International Journal of Management Education, 21(2), 100790. https://doi.org/10.1016/j.ijme.2023.100790
Liu, Y., Zhang, Z., Zhang, W., Yue, S., Zhao, X., Cheng, X., Zhang, Y., & Hu, H. (2023). ArguGPT: evaluating, understanding and identifying argumentative essays generated by GPT models. https://doi.org/10.48550/arxiv.2304.07666
Lodge, J. M. (2024). The evolving risk to academic integrity posed by generative artificial intelligence: Options for immediate action.
Luckin, R. (2018). Machine learning and human intelligence : the future of education for the 21st century. UCL Institute of Education Press.
Lytras, M. D., Alkhaldi, A., Malik, S., Serban, A. C., & Aldosemani, T. (2024). The Artificial Intelligence (AI) Landscape in Higher Education (HE): Current Developments, Opportunities, and Threats. In M. D. Lytras, A. Alkhaldi, S. Malik, A. C. Serban, & T. Aldosemani (Eds.), The Evolution of Artificial Intelligence in Higher Education (pp. 1-10). Emerald Publishing Limited. https://doi.org/10.1108/978-1-83549-486-820241001
Mamabolo, A., & Myres, K. (2019). A detailed guide on converting qualitative data into quantitative entrepreneurial skills survey instrument. Electronic journal of business research methods, 17(3), 102-117. https://doi.org/10.34190/JBRM.17.3.001
Markauskaite, L., Marrone, R., Poquet, O., Knight, S., Martinez-Maldonado, R., Howard, S., Tondeur, J., De Laat, M., Buckingham Shum, S., Gašević, D., & Siemens, G. (2022). Rethinking the entwinement between artificial intelligence and human learning: What capabilities do learners need for a world with AI? Computers and Education: Artificial Intelligence, 3, 100056. https://doi.org/10.1016/j.caeai.2022.100056
Markel, J. M., Opferman, S. G., Landay, J. A., & Piech, C. (2023). GPTeach: Interactive TA Training with GPT-based Students Proceedings of the Tenth ACM Conference on Learning @ Scale, Copenhagen, Denmark. https://doi.org/10.1145/3573051.3593393
Martín-Raugh, M., Kell, H., Ling, G., Fishtein, D., & Yang, Z. (2023). Noncognitive skills and critical thinking predict undergraduate academic performance. Assessment & Evaluation in Higher Education, 48(3), 350-361. https://doi.org/10.1080/02602938.2022.2073964
McGraw, K., & Wong, S. (1996). Forming Inferences About Some Intraclass Correlation Coefficients. Psychological Methods, 1, 30-46. https://doi.org/10.1037/1082-989X.1.1.30
Megahed, F. M., Chen, Y.-J., Ferris, J. A., Knoth, S., & Jones-Farmer, L. A. (2023). How generative AI models such as ChatGPT can be (mis)used in SPC practice, education, and research? An exploratory study. Quality Engineering, 1-29. https://doi.org/10.1080/08982112.2023.2206479
Nguyen, A., Kremantzis, M. D., Essien, A., Petrounias, I., & Hosseini, S. (2024). Enhancing Student Engagement Through Artificial Intelligence (AI): Understanding the Basics, Opportunities, and Challenges. Journal of University Teaching and Learning Practice, 21(6), 1-13. https://doi.org/10.53761/caraaq92
Nikolic, S., Sandison, C., Haque, R., Daniel, S., Grundy, S., Belkina, M., Lyden, S., Hassan, G. M., & Neal, P. (2024). ChatGPT, Copilot, Gemini, SciSpace and Wolfram versus higher education assessments: an updated multi-institutional study of the academic integrity impacts of Generative Artificial Intelligence (GenAI) on assessment, teaching and learning in engineering. Australasian Journal of Engineering Education, 1-28. https://doi.org/10.1080/22054952.2024.2372154
Nyale, D., Karume, S., & Kipkebut, A. (2024). A comprehensive analysis of the role of artificial intelligence in aligning tertiary institutions academic programs to the emerging digital enterprise. Education and Information Technologies. https://doi.org/10.1007/s10639-024-12743-7
Pang, T. Y., Kootsookos, A., & Cheng, C.-T. (2024). Artificial Intelligence Use in Feedback: A Qualitative Analysis. Journal of University Teaching and Learning Practice, 21(06).
Paulsen, M. B. (2013). A Model of Critical Thinking in Higher Education. In (Vol. 30, pp. 41-92). Springer International Publishing AG. https://doi.org/10.1007/978-3-319-12835-1_2
Peres, R., Schreier, M., Schweidel, D., & Sorescu, A. (2023). On ChatGPT and beyond: How generative artificial intelligence may affect research, teaching, and practice. International Journal of Research in Marketing, 40(2), 269-275. https://doi.org/https://doi.org/10.1016/j.ijresmar.2023.03.001
Qadir, J. (2023). Engineering Education in the Era of ChatGPT: Promise and Pitfalls of Generative AI for Education. 2023 IEEE Global Engineering Education Conference (EDUCON), Kuwait, Kuwait., 1-9. https://doi.org/10.1109/EDUCON54358.2023.10125121
Rahman, M. M., & Watanobe, Y. (2023). ChatGPT for Education and Research: Opportunities, Threats, and Strategies. Applied Sciences, 13(9), 5783. https://www.mdpi.com/2076-3417/13/9/5783
Roe, J., Perkins, M., & Ruelle, D. (2024). Is GenAI the Future of Feedback? Understanding Student and Staff Perspectives on AI in Assessment. Intelligent Technologies in Education, Advanced Online Publication. https://doi.org/10.70770/rzzz6y35
Rosenberg, S., Heimler, R., & Morote, E.-S. (2012). Basic employability skills: a triangular design approach. Education & training (London), 54(1), 7-20. https://doi.org/10.1108/00400911211198869
Shively, K., Stith, K. M., & Rubenstein, L. D. (2018). Measuring What Matters: Assessing Creativity, Critical Thinking, and the Design Process. Gifted Child Today, 41(3), 149-158. https://doi.org/10.1177/1076217518768361
Song, C., & Song, Y. (2023). Enhancing academic writing skills and motivation: assessing the efficacy of ChatGPT in AI-assisted language learning for EFL students [Original Research]. Frontiers in Psychology, 14. https://doi.org/10.3389/fpsyg.2023.1260843
Stone, B. W. (2024). Generative AI in Higher Education: Uncertain Students, Ambiguous Use Cases, and Mercenary Perspectives. Teaching of Psychology, 00986283241305398. https://doi.org/10.1177/00986283241305398
Sullivan, M., Kelly, A., & McLaughlan, P. (2023). ChatGPT in higher education: Considerations for academic integrity and student learning. Journal of Applied Learning and Teaching, 6(1).
Tack, A., & Piech, C. (2022). The AI Teacher Test: Measuring the Pedagogical Ability of Blender and GPT-3 in Educational Dialogues. Proceedings of the 15th International Conference on Educational Data Mining, Durham, United Kingdom.
Timmerman, B. E. C., Strickland, D. C., Johnson, R. L., & Payne, J. R. (2011). Development of a ‘universal’ rubric for assessing undergraduates' scientific reasoning skills using scientific writing. Assessment & Evaluation in Higher Education, 36(5), 509-547. https://doi.org/10.1080/02602930903540991
UNESCO. (2023). Education in the age of artificial intelligence, . The UNESCO Courier. https://unesdoc.unesco.org/notice?id=p::usmarcdef_0000387029_eng
Vogt, W. P., Elaine, R. V., Dianne, C. G., & Lynne, M. H. (2014). Selecting the Right Analyses for Your Data : Quantitative, Qualitative, and Mixed Methods [Book]. The Guilford Press. https://search.ebscohost.com/login.aspx?direct=true&db=nlebk&AN=771500&site=ehost-live&authtype=sso&custid=s5445732
Wang, T., Lund, B. D., Marengo, A., Pagano, A., Mannuru, N. R., Teel, Z. A., & Pange, J. (2023). Exploring the Potential Impact of Artificial Intelligence (AI) on International Students in Higher Education: Generative AI, Chatbots, Analytics, and International Student Success. Applied Sciences, 13(11), 6716. https://www.mdpi.com/2076-3417/13/11/6716
Welsh, T. S., & Wright, M. S. (2010). 1 - What is information literacy? In T. S. Welsh & M. S. Wright (Eds.), Information Literacy in the Digital Age (pp. 1-11). Chandos Publishing. https://doi.org/https://doi.org/10.1016/B978-1-84334-515-2.50001-9
Yilmaz, R., & Karaoglan Yilmaz, F. G. (2023). The effect of generative artificial intelligence (AI)-based tool use on students' computational thinking skills, programming self-efficacy and motivation. Computers and Education: Artificial Intelligence, 4, 100147. https://doi.org/https://doi.org/10.1016/j.caeai.2023.100147
Zhao, R., Yunus, M. M., & M. Rafiq, K. R. (2023). The Impact of the Use of ChatGPT in Enhancing Students' Engagement and Learning Outcomes in Higher Education: A Review. International journal of academic research in business and social sciences, 13(12). https://doi.org/10.6007/IJARBSS/v13-i12/20258
Zohny, H., McMillan, J., & King, M. (2023). Ethics of generative AI. Journal of medical ethics, 49(2), 79-80. https://doi.org/10.1136/jme-2023-108909
Downloads
Published
Issue
Section
License
Copyright (c) 2024 Toh Yen Pang, Alexandra Kootsookos, Ben Cheng (Author)
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
