Learning and AI Evaluation of Tutors Responding to Students Engaging in Negative Self-Talk

EasyChair Preprint no. 13351

Abstract

Addressing negative self-talk by students, such as responding to a student when saying, “I am dumb” or “I can’t do this” can be difficult for even the most experienced tutor. Despite potential tutor learning from scenario-based lessons on this topic, human-graded assessment remains time-consuming. Leveraging generative AI for evaluating textual responses in online training presents a scalable solution. Research suggests a tutor validates student’s feelings when they speak negatively of themselves, e.g., by a tutor responding, “I understand how you feel” or “I recognize this is difficult.” This ongoing work assesses the performance of 60 undergraduate tutors within an online lesson on enhancing tutors’ abilities to respond to students engaging in negative self-talk. We find statistically significant tutor learning gains from pretest to posttest. Additionally, we describe a method of using generative AI for assessing tutors’ responses to predict the best approach and subsequently explain the rationale behind it. Using the large language model GPT-4, we find high absolute performance when evaluating tutor responses involving predicting (F1 = 0.85) and explaining (F1 = 0.83) the best approach. Minor improvements are needed to the lesson itself. A future goal of this work is to fully develop automated systems of assessing tutor learning attending to barriers to students’ motivation and doing so at scale.

Keyphrases: assessment, Generative AI, large language models, Prompt Engineering, tutor training

@Booklet{EasyChair:13351,
  author = {Danielle R Thomas and Jionghao Lin and Shambhavi Bhushan and Ralph Abboud and Erin Gatz and Shivang Gupta and Kenneth Koedinger},
  title = {Learning and AI Evaluation of Tutors Responding to Students Engaging in Negative Self-Talk},
  howpublished = {EasyChair Preprint no. 13351},

  year = {EasyChair, 2024}}