Virtual School Meanderings

October 19, 2016

AECT 2016 – Educational Data Mining in Program Evaluation: Lessons Learned

This is the third session that I am blogging from the Association for Educational Technology and Communications 2016 annual convention.

Educational Data Mining in Program Evaluation: Lessons Learned

  • In Event: DDL – Data Mining & Program Evaluation

Wed, Oct 19, 9:15 to 10:15am, Conf Ctr, Pavilion 10

Short Description

Researchers present findings from a series of data mining studies, primarily examining data mining as part of an innovative triangulated approach in program evaluation. Findings suggest that is it possible to apply EDM techniques in online and blended learning classrooms to identify key variables important to the success of learners. Lessons learned will be shared as well as areas for improving data collection in learning management systems for meaningful analysis and visualization.


  • Kerry Rice, Boise State University
  • JUI-LONG HUNG, Boise State University
  • Yu-Chang Hsu, Boise State University
  • Brett Shelton, Boise State University

Kerry mainly led this session.  It focused on a series of studies that used data mining – many of which were program evaluations – to highlight the process that they have used as a part of this line of inquiry.  Kerry began the session with a quick overview of the xxx studies (and I have captured an image of the slide that had all of the formal article citations below).


There were several K-12 online and blended learning studies that were included in the slides (and I tried to capture those specific slides – see below).



Kerry then transitioned to looking at the topic of what is engagement – which, in the end, they operationalized variables that focused on where the student was interacting with the course in some way shape or form (e.g., clicks in the course, logins, discussion posts read, etc.).  But the slide that she had posted had some two to three dozen individual variables taken from the learning management system and survey data.

Yu-Chang then took over and discussed the process of cluster analysis, which at its basic form is using a statistical procedure to figure out what series of variables – often in what sequence – can be used to accurately predict student success and student failure.  For example, average time spent, average dys participated, average frequency of mouse clicks, average time spent per session, and average frequency of mouse clicks per session.  I have posted the actual analysis slide below.


There was a specific slide that focused on one of the K-12 online or blended learning studies, which I wasn’t able to capture an image.

Kerry then took over again and described some of the sequential association or path analysis that they had conducted.  Essentially, what paths do students who have success take compared to paths students who did not have success.  Interestingly, they found that the path didn’t tell them much, but certain actions did.  For example, posting a discussion entry was a useful predictor.

In the summary slides, Kerry did note a couple of relevant K-12 findings:

  • advanced courses – high engagement and high performance
  • entry level course – low performance regardless of engagement
  • successful students: female, younger, were more engaged in their courses
  • at-risk students: male, older, took entry-level courses

As an aside, Kerry mentioned an article by Ryan Baker that looked at the issue of “time on task.”  I think it may be this one:

Kovanovic, V. Gasevic, D., Dawson, S., Joksimovic, S., Baker, R.S., Hatala, M. (2016) Does Time-on-task Estimation Matter? Implications on Validity of Learning Analytics Findings. Journal of Learning Analytics, 2 (3), 81-110 Retrieved from

Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

Blog at

%d bloggers like this: