Evaluation of the AI-Teacher Teaching Tasks Spectrum via Practitioner Review
DOI:
https://doi.org/10.70770/rsxpge96Keywords:
AI in education, educational technology, Teacher-AI Collaboration, AI-Teacher Pedagogical FrameworksAbstract
This pilot study aims to evaluate the AI-Teacher Teaching Task Spectrum (AITTTS), a framework designed to categorise human-AI intervention levels based on the teaching tasks and their suitability for AI or human intervention. The primary objective was to provide preliminary validation of the framework’s practical utility by examining its alignment with current literature and gathering practitioner feedback. A systematic literature review was conducted, focusing on three key studies that offered insights into AI-teacher task delegation. Additionally, a structured survey was used to collect data from three expert practitioners in AI and education, with Fleiss’ Kappa applied to measure agreement. The findings indicated substantial agreement (Fleiss’ κ = 0.73) on the framework’s validity, particularly for identifying tasks suitable for AI, such as procedural and knowledge-based activities. However, the study’s small sample size and limited geographic diversity restrict the generalisability of the findings. Disagreements, especially regarding AI’s role in creative and relational tasks, highlight areas requiring further exploration. As the first step in an iterative research agenda, this pilot study provides foundational insights into the framework’s potential. Future research will involve expanded participant samples, broader educational contexts, and iterative refinements to enhance the framework’s applicability and generalisability across varied educational settings.
Downloads
References
Alharmoodi, B. Y. R., & Lakulu, M. M. (2022). The Formulation and Validation of a Conceptual Framework for the Transition from E-government to M-government. European Journal of Interdisciplinary Studies, 8(1), 23–34. https://doi.org/10.26417/502cxc20
Ashri, D., & Sahoo, B. P. (2021). Open Book Examination and Higher Education During COVID-19: Case of University of Delhi. Journal of Educational Technology Systems, 50(1), 73–86. https://doi.org/10.1177/0047239521013783
Bradford, N., Chambers, S., Hudson, A., Jauncey-Cooke, J., Penny, R., Windsor, C., & Yates, P. (2019). Evaluation frameworks in health services: An integrative review of use, attributes and elements. Journal of clinical nursing, 28(13-14), 2486–2498. https://doi.org/10.1111/jocn.14842
Cotton, D. R. E., Cotton, P. A., & Shipway, J. R. (2023). Chatting and cheating: Ensuring academic integrity in the era of ChatGPT. Innovations in Education and Teaching International, 1–12. https://doi.org/10.1080/14703297.2023.2190148
Dawadi, S., Shrestha, S., & Giri, R. A. (2021). Mixed-Methods Research: A Discussion on its Types, Challenges, and Criticisms. Journal of Practical Studies in Education, 2(2), 25-36 DOI: https://doi.org/10.46809/jpse.v2i2.20
De Freitas, J., Uğuralp, A. K., Oğuz‐Uğuralp, Z., & Puntoni, S. (2023). Chatbots and mental health: Insights into the safety of generative AI. Journal of Consumer Psychology. https://doi.org/10.1002/jcpy.1393
Dumas, J., Sorce, J., & Virzi, R. (1995). Expert Reviews: How Many Experts is Enough? Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 39(4), 228-232. https://doi.org/10.1177/154193129503900402
Fernández-Gómez, E., Martín-Salvador, A., Luque-Vara, T., Sánchez-Ojeda, M. A., Navarro-Prado, S., & Enrique-Mirón, C. (2020). Content Validation through Expert Judgement of an Instrument on the Nutritional Knowledge, Beliefs, and Habits of Pregnant Women. Nutrients, 12(4), 1136. https://doi.org/10.3390/nu12041136
Friedman, S., Hurley, T., & Fishman, T. (2020). COVID-19’s impact on higher education: Strategies for tackling the financial challenges facing colleges and universities. https://doi.org/https://www2.deloitte.com/content/dam/Deloitte/us/Documents/public-sector/us-gps-covid-university-finance.pdf
Fölstad, A., Araujo, T., Law, E. L., Brandtzæg, P. B., Papadopoulos, S., Reis, L., Báez, M., Laban, G., McAllister, P., Ischen, C., Wald, R., Catania, F., Von Wolff, R. M., Hobert, S., & Luger, E. (2021). Future directions for chatbot research: an interdisciplinary research agenda. Computing, 103(12), 2915–2942. https://doi.org/10.1007/s00607-021-01016-7
Gerring, J. (2016). Case study research. https://doi.org/10.1017/9781316848593
Hasanpoor, E., Hallajzadeh, J., Siraneh, Y., Hasanzadeh, E., & Haghgoshayie, E. (2019). Using the methodology of systematic review of reviews for Evidence-Based Medicine. Ethiopian Journal of Health Sciences, 29(6). https://doi.org/10.4314/ejhs.v29i6.15
Hennink, M., & Kaiser, B. N. (2022). Sample sizes for saturation in qualitative research: A systematic review of empirical tests. Social Science & Medicine, 292, 114523. https://doi.org/10.1016/j.socscimed.2021.114523
Holmes, W., Bialik, M., & Fadel, C. (2019). Artificial intelligence in education: Promises and Implications for Teaching and Learning.
Koh, J., Cowling, M., Jha, M., & Sim, K. N. (2022). Collaborating with AIed for better student-teacher reconnection. ASCILITE Publications, e22126. https://doi.org/10.14742/apubs.2022.126
Koh, J., Cowling, M., Jha, M., & Sim, K. N. (2023). The Human Teacher, the AI Teacher and the AIed-Teacher Relationship. Journal of Higher Education Theory and Practice, 23(17). https://doi.org/10.33423/jhetp.v23i17.6543
Leung L. Validity, reliability, and generalizability in qualitative research. J Family Med Prim Care. 2015 Jul-Sep;4(3):324-7. doi: 10.4103/2249-4863.161306. PMID: 26288766; PMCID: PMC4535087.
Lodge, J. M., Thompson, K., & Corrin, L. (2023). Mapping out a research agenda for generative artificial intelligence in tertiary education. Australasian Journal of Educational Technology, 39(1), 1–8. https://doi.org/10.14742/ajet.8695
Luckin, R., Holmes, W., Griffiths, M., & Forcier, L. B. (2016). Intelligence unleashed: An argument for AI in education. Pearson.
McHugh M. L. (2012). Interrater reliability: the kappa statistic. Biochemia medica, 22(3), 276–282. https://pubmed.ncbi.nlm.nih.gov/23092060/
Nowell, L., Norris, J. M., White, D., & Moules, N. J. (2017). Thematic analysis. International Journal of Qualitative Methods, 16(1), 160940691773384. https://doi.org/10.1177/1609406917733847
Palinkas, L. A., Mendon, S. J., & Hamilton, A. B. (2019). Innovations in mixed methods evaluations. Annual Review of Public Health, 40(1), 423–442. https://doi.org/10.1146/annurev-publhealth-040218-044215
Pérez, J. Q., Daradoumis, Τ., & Puig, J. M. M. (2020). Rediscovering the use of chatbots in education: A systematic literature review. Computer Applications in Engineering Education, 28(6), 1549–1565. https://doi.org/10.1002/cae.22326
Polit, D. F., & Beck, C. T. (2021). Nursing research: Generating and Assessing Evidence for Nursing Practice. LWW.
Selwyn, N. (2019). Should robots replace teachers?: AI and the Future of Education. Polity.
Seo, K., Tang, J., Roll, I., Fels, S., & Yoon, D. (2021). The impact of artificial intelligence on learner–instructor interaction in online learning. International Journal of Educational Technology in Higher Education, 18(1), 54. https://doi.org/10.1186/s41239-021-00292-9
Shen, L., & Su, A. (2020). The Changing Roles of Teachers With AI (pp. 1–25). https://doi.org/10.4018/978-1-5225-7793-5.ch001
Tiddi, I., & Schlobach, S. (2022). Knowledge graphs as tools for explainable machine learning: A survey. Artificial Intelligence, 302, 103627. https://doi.org/10.1016/j.artint.2021.103627
Tufford, L., & Newman, P. (2010). Bracketing in qualitative research. Qualitative Social Work, 11(1), 80–96. https://doi.org/10.1177/1473325010368316
Vasileiou, K., Barnett, J., Thorpe, S., & Young, T. (2018). Characterising and justifying sample size sufficiency in interview-based studies: systematic analysis of qualitative health research over a 15-year period. BMC Medical Research Methodology, 18(1). https://doi.org/10.1186/s12874-018-0594-7
Wang, S., Wang, F., Zhu, Z., Wang, J., Tran, T., & Du, Z. (2024). Artificial intelligence in education: A systematic literature review. Expert Systems With Applications, 252, 124167. https://doi.org/10.1016/j.eswa.2024.124167
Williamson, B., Eynon, R., & Potter, J. (2020). Pandemic politics, pedagogies and practices: digital technologies and distance education during the coronavirus emergency. Learning Media and Technology, 45(2), 107–114. https://doi.org/10.1080/17439884.2020.1761641
Weyant, E. (2022). Research Design: Qualitative, Quantitative, and Mixed Methods Approaches, 5th edition. Journal of Electronic Resources in Medical Libraries, 19(1–2), 54–55. https://doi.org/10.1080/15424065.2022.2046231
Young, W. (2022). Virtual Pastor: virtualization, AI, and pastoral care. Theology and Science, 20(1), 6–22. https://doi.org/10.1080/14746700.2021.2012915
Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education – where are the educators? In International Journal of Educational Technology in Higher Education (Vol. 16, Issue 1). Springer Netherlands. https://doi.org/10.1186/s41239-019-0171-0
Downloads
Published
Issue
Section
License
Copyright (c) 2024 Josiah Koh, Michael Cowling, Meena Jha, Kwong Nui Sim (Author)
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
