For the past ten years, Meredith Mountford, an associate professor in the Department of Educational Leadership and Research Methodologyat Florida Atlantic University, has been requiring her doctoral students to read NEPC’s expert, third-party reviews of selected, non-peer-reviewed publications. This school year, for the first time, Mountford tried something new: She asked the students in her Seminar in Administration course to write their own reviews. This week, NEPC is publishing one of those reviews as part of its Reviews Worth Sharingfeature. Although not commissioned or edited by NEPC, these reviews are published because they contribute to the goal of helping policymakers, reporters, and others assess the social science merits of reports and judge their value in guiding policy. The review that emerged from Mountford’s class is written by Dustin Pappas. Pappas’ piece examines a March 2018 Heritage Foundation report, Focusing on School Safety After Parkland. In the Q & A below, Mountford explains how she came to assign her students to write their own NEPC-style reviews and what she believes they learned from the assignment.
Note: This interview has been edited and condensed for clarity.
Q: What made you start assigning NEPC-style reviews?
A: I noticed that when I assigned students to read a report from a far left or right foundation or organization, and asked for a “critical response”, I often just got a review or summary of the report. I wanted to students to be critical and suspicious of studies emanating from large organizations that had some political clout and those that had agendas that did not prioritize public education. I would point out to students that these were some of the same “studies” picked up by popular news magazines and newspapers, like Time Magazine or USA Today. This lent credibility to studies that were not empirically rigorous or methodologically sound and therefore undermined efforts to improve public education. Ultimately, I want to teach students to detect shortcomings in a heartbeat as well as explain insufficiencies to others. School leaders cannot and should not be duped by reports and whitepapers that propose silver bullets and it’s important they can communicate this to staff and faculty.
Q: What kind of reactions have you received from students about the assignment?
A: I had 12 students enrolled originally, but by the third class, I was down to nine which is not that unusual for a PhD level course. I think those who didn’t return may have found the syllabus a bit intimidating. Later, I heard from several students who stuck it out for the entire course that they were initially intimidated by the syllabus. The nine who stayed in the course were very positive on the course evaluation. Some inquired as to why this class wasn’t required in the major discipline at the PhD level. They seemed very engaged with the topics they chose and intrigued with the concept of being the “expert” critiquing research.
Q: How did you frame the assignment for your students?
A: I had them use the reviews on the NEPC website as the model for the format in which to write the critiques. I spent the first half of the semester teaching students what to look for when critiquing a report, study, or whitepaper. We did several together until I felt they were ready to pick out one of their own issues, to find a report on it, and write a critique. Initially, they were bumpy, but with the help of the NEPC website and Fred Pyrzcak’s text Evaluating research in academic journals: A practical guide to realistic evaluation, I began receiving some incredibly thoughtful reviews from students.
Q: Tell me something about the students who took the course.
A: The majority of students in the course were high-level leaders in some of the largest urban school districts in the country. For these students to feel as though they had a chance to publicly critique (or set the record straight) regarding the misinformation they had encountered leading their districts was quite alluring. Out of the nine students, at least five wrote reviews that were strong enough that, with some work, they are potentially publishable in a wide range of outlets.
Q: How many reviews have students written since you started assigning them?
A: They wrote three for this class and the PhD students who enroll in the Ethics and Policy Alternatives course I will teach in the fall will also be assigned three reviews.
Q: What are some examples of the topics of the reports that students wrote about?
A: Report topics included technology, ESL, community outreach programs for Hispanics, leader evaluations, and teacher evaluations.
Q: What do you think students learned from the assignment?
A: I think they learned to distinguish rhetoric from reality. They learned how to critique research that is less than reliable and to communicate its deficiencies to colleagues in a useful way that inspires change. They gained the ability to distinguish between problematic think tank reports and data from rigorous research. Effective school leaders today must lead on their toes. I hope this class helps them to do that. I constantly remind them that they too must always be open to criticism of their own research. I think the review assignment may help them when they write their dissertations because they sometimes grow overly attached to their proposed methodologies and getting them to change can be very difficult. Now if their own dissertation methodologies are flawed, they may have a better appreciation for the need for reviews and corrections.
Subscribe: Feedly
12 Unique Blogs Are Written By Professors
