EliRank: A Code Editing History Based Ranking Model for Early Detection of Students in Need

Abstract

Research on programming education shows that novice programming students benefit significantly from one-to-one tutoring. While many systems propose to replicate the effectiveness of one-to-one tutoring in large-scale classes, it remains a challenge to develop systems with an approach to finding students who need the tutors' help the most. In this paper, we explore the idea of predicting the priority of students in need with a data-driven approach. Among various metrics to calculate the priority of students in need, we adopt time-on-task metric. Previous studies have found that excessively long time-on-task can be used as an indication of students' struggling. Aligned with this, we reduce the problem of finding students with the highest priority to the problem of finding students with the longest time-on-task. To solve the reduced problem, we present EliRank, a ranking model that finds students with the longest estimated time-on-task, using the students' first few minutes of fine-grained code editing history. EliRank recommends students in the descending order of estimated time-on-task, enabling tutors to efficiently monitor and find the students in need at scale in real time. To evaluate the performance of EliRank, we build and publish a new real-world dataset consisting of 15 programming exercises solved by 4000+ students in an introduction to programming class at a university. Unlike the currently available open code editing history datasets, our dataset contains code editing operations at a character-level granularity to minimize the loss of contextual information from students. We also introduce diff-augmented abstract syntax tree (DAST), a novel structured code representation that minimizes the loss of fine-grained code change information during code parsing. The evaluation of EliRank on our dataset shows that EliRank effectively finds students with the longest estimated time-on-task, for early detection of students in need. Also, we illustrate in depth (i) the effectiveness of DAST, (ii) the potential to control the tradeoff between early detection and the prediction accuracy of the model, and (iii) the transferability to unseen programming exercise via zero-shot transfer learning.