EDTECH597 - Commentary Entry: HMH Fuse Pilot Study Will Fissle

I’ve been following this HMH Fuse project for a while now (see Mobile Learning – HMH Fuse: Algebra I).  As someone with an interest in the use of mobile devices at the K-12 level, and particularly with their potential application for K-12 online learning, I’m very interested in the potential of this electronic book they’ve developed for the iPad.  A while back I was looking for any written material about the project and I came upon the Press for HMH Fuse page.  Being a researcher, there were a couple of stories that caught my attention, the first of which was:

Algebra, Meet the iPad: A Year-Long Study Explores Learning With the Tablet

In reading this article, one of the things I learned was that Empirical Education was the evaluator for this project.  In reviewing their website, it appears that Empirical Education, which is essentially a testing company, has a fair degree of evaluation experience with mathematics initiatives (including several National Science Foundation and other federally funded projects).  In looking through their personnel, there are a lot of statistics or mathematics folks, some educators (although they all seem to have a background in statistics or mathematics as well).

Anyway, the second article that caught my attention was:

HMH Fuse and the future of K-12 mobile education on the iPad

I noticed this one because it gave a fairly good description of the actual study:

“The Pilot Project

HMH bought and donated 400 64 GB Wi-Fi+3G iPads to schools in the Long Beach, Riverside, Fresno and San Francisco unified school districts. Eighth grade students in the test classes were loaned the iPads. Another 600 students took classes using only the textbooks. The same teachers taught both the test and control classes.

There was no possibility of front-loading classes with the better students getting the iPads since HMH had no control over which classes would be using the textbook or the app. The makeup of classes were determined as they always had been — through standard class rosters provided by the school districts. The teachers who volunteered to take part knew that this would be more work than usual, and the technical capabilities of these teachers varied widely — most of them had never touched an iPad. One day of iPad training was provided by HMH; some teachers had as little as three hours of instruction, and then HMH went away. The only contact HMH had with the schools and teachers was to provide technical assistance. Assistance was also available from representatives in the Apple educational division.

Empirical Education, a large testing company, was employed to conduct the study; it started in September of last year and will finish at the end of the school year this June when data will be compiled and presented to HMH. The major intentions were to see how the impact of multimedia and multiple presentation methods affected student perception, achievement and usage trends.

Outside of test scores, Empirical collected information on topics such as how many times students watched a video or took a quiz. Teachers were asked to periodically complete qualitative surveys on their reactions to the app, with questions ranging from how using the app affected their teaching to their impressions of student reactions. The main thrust, however, was on the students who comprise the target market.

The study began with all 1000 students using the textbook for one month since the HMH Fuse Algebra app wasn’t completed until early October. All the content, including multimedia components, was available to the textbook classes on a site provided by HMH, and textbook class teachers were allowed to use all of these assets by projecting them to their classes as often as they liked. Just as students get to take home their textbooks for homework, the iPads were taken home every night by members of the HMH Fuse classes. It was left up to the teacher whether to lock down the iPads and only allow use of HMH Fuse, or to open them up to be general-purpose devices.

After tracking site usage and finding that students weren’t visiting anywhere inappropriate, most teachers left the iPads open, and students were encouraged to download apps and music for their personal use. This was often decided after discussions took place with teachers, parents and students. A component of the study was a student’s sense of ownership of the iPad.

Each iPad contained a unique identifier that was mapped to an individual student. When a test or quiz was taken, the results were immediately transferred to a teacher reporting site that provided not only the grades for a quiz or test, but also how the students fared on each relevant skill set in the assessment. In addition to assisting the teachers, this data was also harvested for the pilot study.”

Okay, take a minute to read that again.  You have 600 students that are using textbooks and 400 students using iPads.  There isn’t random assignment of the students, simply one group of classes gets iPads and and other groups of classes don’t.  This makes the assumption that the scheduling of these students equals random assignment and should normalize the students, but we all know that isn’t the case.  The students who register for Algebra I who also sign up for a basic English course, physical education, shop, and environmental science (or some other basic level science) are all going to end up in the same Algebra I class.  Similarly, the students who all signed up for band, chemistry, AP history, and honors English will also likely end up in the same Algebra I class.  That’s how high school scheduling works, higher ability students are likely to take the same kinds of courses and have a greater likelihood that in order to accommodate all of their selections they end up in many of the same classes together.  Similarly, the lower ability students are likely to take the same kinds of courses and end up in many of the same classes together.

Now what happens if the class with the higher ability students are the ones that end up with the iPads and the lower ability students are using the textbooks?  Can we really say that any difference in performance between the two groups was caused by the presence of the iPad or the use of the HMH Fuse program?

To further complicate issues, if you go back to the Algebra, Meet the iPad: A Year-Long Study Explores Learning With the Tablet article and read down a bit further, you are told:

Q. How will the iPad-taught class different from a traditional algebra class?

What we’ve seen in practice is the fact that it’s bringing everything to one place that’s making it exciting. The convenience factor, the simplicity factor — that’s revolutionary. For example, if you’re working through a lesson, there are three or four algorithms presented. With a textbook, if you want to learn more about one of the examples, you have to stop looking in your book and go online to our website and navigate that particular section and view our video there.

Instead, on the iPad, you simply click on “view video” and up comes our professor, Dr. Edward Burger, the Bill Nye of math education. Students have written to him saying he’s changed their opinion of what math is. So to have him right there, you can see how it’s natural for students to tap “view video,” as opposed to setting their book down and going to the computer.

Another example is, when students are working on a problem, they can simply click on “check answer,” and up comes, “that’s correct, and here’s why,” or “that’s incorrect, and here’s why.” As opposed to when they’re working on paper or even online, those pieces are a little more drawn out.

So this is not a study of whether the use of an electronic textbook in the form of HMH Fuse is more effective than a traditional mathematics textbook.  It can’t be given the fact that in addition to the shiny new device; in addition to the electronic, interactive textbook; there are also substantial differences in the pedagogy being employed by the teachers who have the iPads compared to the teachers who are using the textbooks.  It doesn’t matter if it is the same group of teachers for both students, if the methods of instruction are different can we really say that HMH Fuse had anything to do with any differences in student performance?

The way this whole pilot project is set-up reminds me of Clark’s infamous line about technology affects student performance as much as the delivery truck affects the nutritional value of the groceries it carries.

If I look into my crystal ball I can already tell you that this paid evaluation will likely find exactly what the textbook publishing company wants to find, students perform better when using HMH Fuse and students are more motivated and engaged when using HMH Fuse.  The fact of the matter is because of a poor research design and a misunderstanding that you cannot attribute any finding simply to the presence or absence of technology will render these results unreliable and invalid.  Not that the public will ever hear about it though…  Let’s face it, HMH has a healthy marketing budget!

Just so you don’t think I’m totally down on this program, as I’m not (I actually think it looks rather good, it is just the faulty research I’m down on), I wanted to pass on this positive review of HMH Fuse.

About these ads