This is the third session that I am blogging from the Association for Educational Technology and Communications 2016 annual convention.
Educational Data Mining in Program Evaluation: Lessons Learned
- In Event: DDL – Data Mining & Program Evaluation
Wed, Oct 19, 9:15 to 10:15am, Conf Ctr, Pavilion 10
Researchers present findings from a series of data mining studies, primarily examining data mining as part of an innovative triangulated approach in program evaluation. Findings suggest that is it possible to apply EDM techniques in online and blended learning classrooms to identify key variables important to the success of learners. Lessons learned will be shared as well as areas for improving data collection in learning management systems for meaningful analysis and visualization.
- Kerry Rice, Boise State University
- JUI-LONG HUNG, Boise State University
- Yu-Chang Hsu, Boise State University
- Brett Shelton, Boise State University
Kerry mainly led this session. It focused on a series of studies that used data mining – many of which were program evaluations – to highlight the process that they have used as a part of this line of inquiry. Kerry began the session with a quick overview of the xxx studies (and I have captured an image of the slide that had all of the formal article citations below).
There were several K-12 online and blended learning studies that were included in the slides (and I tried to capture those specific slides – see below).
Kerry then transitioned to looking at the topic of what is engagement – which, in the end, they operationalized variables that focused on where the student was interacting with the course in some way shape or form (e.g., clicks in the course, logins, discussion posts read, etc.). But the slide that she had posted had some two to three dozen individual variables taken from the learning management system and survey data.
Yu-Chang then took over and discussed the process of cluster analysis, which at its basic form is using a statistical procedure to figure out what series of variables – often in what sequence – can be used to accurately predict student success and student failure. For example, average time spent, average dys participated, average frequency of mouse clicks, average time spent per session, and average frequency of mouse clicks per session. I have posted the actual analysis slide below.
There was a specific slide that focused on one of the K-12 online or blended learning studies, which I wasn’t able to capture an image.
Kerry then took over again and described some of the sequential association or path analysis that they had conducted. Essentially, what paths do students who have success take compared to paths students who did not have success. Interestingly, they found that the path didn’t tell them much, but certain actions did. For example, posting a discussion entry was a useful predictor.
In the summary slides, Kerry did note a couple of relevant K-12 findings:
- advanced courses – high engagement and high performance
- entry level course – low performance regardless of engagement
- successful students: female, younger, were more engaged in their courses
- at-risk students: male, older, took entry-level courses
As an aside, Kerry mentioned an article by Ryan Baker that looked at the issue of “time on task.” I think it may be this one:
Kovanovic, V. Gasevic, D., Dawson, S., Joksimovic, S., Baker, R.S., Hatala, M. (2016) Does Time-on-task Estimation Matter? Implications on Validity of Learning Analytics Findings. Journal of Learning Analytics, 2 (3), 81-110 Retrieved from http://www.columbia.edu/~rsb2162/JLA-Vita.pdf