New Release: What Works Clearinghouse Procedures and Standards Handbook, Version 5.0

While I’ve expressed my concerns about the nature of the “gold standard” required for inclusion in the WWC (see here and here), it is always important to pass this information along.

The What Works Clearinghouse is excited to announce the release What Works Clearinghouse Procedures and Standards Handbook, Version 5.0. The newest version of the Handbook added a new chapter that provides a succinct summary of the WWC and is intended for a broad audience of practitioners, policymakers, and researchers.  It also has improved organization to align the flow of content with the study review process. Learn more about the updated Handbook at a webinar on August 25 at 2pm ET. Major changes to the procedures and standards include: The WWC no longer allows for topic-specific customization of the standards and all WWC study reviews will be conducted using the Study Review Protocol. The WWC aligned its effectiveness ratings with U.S. Department of Education evidence definitions for individual studies and synthesis products. The WWC now determines effectiveness ratings at the outcome domain level because a synthesis of multiple findings in the same domain will tend to provide a better representation of the underlying construct than any single measurement. Effectiveness ratings for individual study reviews will be based on outcome measures that are independent of intervention developers and study authors. The WWC will attempt to ensure that a majority of the meta-analytic weight is based on findings rated Meets WWC Standards Without Reservations when a cross-study synthesis includes findings rated Meets WWC Standards Without Reservations and Meets WWC Standards With Reservations and the sample size is sufficiently large. High attrition randomized controlled trials and high attrition regression discontinuity design studies no longer need to demonstrate baseline equivalence to be rated Meets WWC Standards With Reservations when attrition bias is assessed using the optimistic boundary. Teams conducting WWC reviews are responsible for determining whether to use an optimistic or a cautious attrition boundary for a specific review and for documenting their reasoning based on the principles described in the Handbooks. The WWC has removed procedures for WWC-applied difference-in-difference adjustments. The WWC no longer considers bundled—or combined—interventions a confounding factor in reviews of individual studies because a bundled intervention can produce a valid impact estimate for the “package” of interventions. The WWC now classifies cluster randomized control trials as having either a low or high risk of bias due to compositional change from joiners. Single-case design (SCD) studies that use multiple baseline/multiple probe, treatment reversal, and changing criterion designs need to have at least six data points in the initial baseline phases for their findings to be eligible for the rating Meets WWC Standards Without Reservations. There is also an exception for minimum data point requirements. The WWC modified the interobserver agreement requirements for SCDs to apply to all data in the study. The WWC added a new “limit risk of bias” step to the review process for multiple baseline/multiple probe, treatment reversal, and changing criterion SCDs that are eligible for the rating Meets WWC Standards Without Reservations. The WWC provided updated guidance on how to rate SCDs with features from multiple design types and SCDs with more cases and/or phases than the minimum required to meet WWC standards. The Institute of Education Sciences, a part of the U.S. Department of Education, is the nation’s leading source for rigorous, independent education research, evaluation, statistics, and assessment.

By visiting Newsflash you may also sign up to receive information from IES and its four Centers NCES, NCER, NCEE, & NCSER to stay abreast of all activities within the Institute of Education Sciences (IES).