Virtual School Meanderings

April 25, 2022

AERA 2022 – A Novel Adoption of Two Online Teaching Self-Evaluation Instruments Among a Public Pre-K–12 Teacher Sample

The twenty-seventh of the K-12 Online Learning sessions from the 2022 annual meeting of the American Educational Research Association that I am blogging is:

A Novel Adoption of Two Online Teaching Self-Evaluation Instruments Among a Public Pre-K–12 Teacher Sample

  • In Event: Supporting K–12 Teaching and Learning With Online Resources and Tools

Mon, April 25, 8:00 to 9:30am PDT (8:00 to 9:30am PDT), SIG Virtual Rooms, SIG-Online Teaching and Learning Virtual Paper Session Room

Abstract

As the COVID-19 pandemic forced schools to close their buildings, public PK12 educators in the US had to pivot to something they had never done before or been trained to do: teach online. While significant research exists on how educators learn to teach online in higher education and specialty PK12 settings, the pandemic provided a novel opportunity to explore how the general teaching population feels about and learns to teach online. We present the novel adoption of two online teaching self-evaluation instruments and initial findings from an ongoing study of PK12 teachers across 10 public school districts in the MidAtlantic US. Survey data sheds light on how to measure teachers’ self-evaluations of their online teaching strategies and self-efficacy.

Authors

  • Virginia Byrne, Morgan State University
  • Diane Jass Ketelhut, University of Maryland – College Park

Which was part of this larger session:

Supporting K–12 Teaching and Learning With Online Resources and Tools

Mon, April 25, 8:00 to 9:30am PDT (8:00 to 9:30am PDT), SIG Virtual Rooms, SIG-Online Teaching and Learning Virtual Paper Session RoomSession Type: Paper Session

Abstract

Preparation and support for K-12 teaching with online resources and tools is a growing area of interest. Supporting K-12 learners is also garnering increasing interest. This session will feature research research about K-12 online teaching and learning. Presentations feature international work as well as work from the United States.

Sub Unit

  • SIG-Online Teaching and Learning

Chair

  • Jacqueline Zweig, Education Development Center, Inc.

Papers

  • Online Teaching in K–12 Education: A Systematic Review – Carla C. Johnson, North Carolina State University; Janet Walton, North Carolina State University; Jennifer Brammer Elliott, North Carolina State University; Lacey Jean Strickler Eppard, University of Toledo
  • Timing of Enrollment and Online Course Completion – Jacqueline Zweig, Education Development Center, Inc.; Erin Stafford, Education Development Center, Inc.
  • A Novel Adoption of Two Online Teaching Self-Evaluation Instruments Among a Public Pre-K–12 Teacher Sample – Virginia Byrne, Morgan State University; Diane Jass Ketelhut, University of Maryland – College Park
  • Factors Influencing Chinese K–12 Teachers’ Intention to Teach Online During the Pandemic – Yu Qing, East China Normal University
  • Homeschooling in Norway During the COVID-19 Pandemic: Unequal Opportunities and Little Digital Innovation – Cecilie Pedersen Dalland, OsloMet – Oslo Metropolitan University; Marte Blikstad-Balas, University of Oslo; Kirsti Klette, University of Oslo; Astrid Roe, University of Oslo

Discussant

  • Mary F. Rice, The University of New Mexico

The presenters were an external evaluation team during the 2020-21 school year to work with 11 school districts to improve virtual teaching.  The talk today focused mainly on two instruments – one they created called the Self-Evaluation of Online Teaching for PK12 and then a “novel use” of the Michigan Nurse Educators Sense of Efficacy for Online Teaching scale (Robinia, 2008; Robinia & Anderson, 2010).  The context for the study was part of a larger PD program that was focused on online and hybrid instruction in 11 districts in Maryland.  The PD program began in January 2021 and ended in March 2022, and it included webinars, discussions, and meetings – with approximately 450 teachers participating.

The presenters collected data through three surveys.  The first was in January 2021, which had 300 teachers take it.  The second was in June, which had 90 teachers take it.  The third was in March 2022, which had 54 people complete it.  Only 32 teachers took all three surveys.

The first instrument hat was used was entitled the Self-Evaluation of Online Teaching for PK12.  It was based on an instrument developed from this work:

Donlan, A. E., Loughlin, S., & Byrne, V. L. (2019). The fearless teaching framework: A model to synthesize foundational education research for university instructors. To Improve the Academy: A Journal of Educational Development 38(1), 33–49. https://quod.lib.umich.edu/t/tia/17063888.0038.102/–fearless-teaching-framework-a-model-to-synthesize?rgn=main;view=fulltext

Byrne, V. L., & Donlan, A. E. (2020). Presenting a validated mid-semester evaluation of college teaching to improve online teaching. Online Learning24(2), 94-110. https://olj.onlinelearningconsortium.org/index.php/olj/article/view/2126 (table 3 in this document provides the original instrument)

Donlan, A. E., & Byrne, V. L. (2020). Confirming the factor structure of a research-based mid-semester evaluation of college teaching. Journal of Psychoeducational Assessment38(7), 866-881. https://doi.org/10.1177/0734282920903165

Apparently, the presenters revised the language on the classroom climate, active teaching practices, and reliable assessments sections of this instrument to make the language more K-12 friendly.  For example, this was one of the changes:

Original – My instructor creates an online classroom that is supportive for learning.

K-12 Revision – I have created an online classroom that is supportive for learning.

The presenters also removed the relevant course content section altogether.

The second instrument was the Michigan Nurse Educators Sense of Efficacy for Online Teaching scale, which was originally developed by:

Robinia, K. A. (2008). Online teaching self-efficacy of nurse faculty teaching in public, accredited nursing programs in the state of Michigan. [Doctoral Dissertation] Western Michigan University. https://www.proquest.com/docview/304445650?pq-origsite=gscholar&fromopenview=true (Appendix I on page 156 contains a copy of the instrument)

It appears that this instrument was used verbatim.

The results revealed that the Self-Evaluation of Online Teaching for PK12 had good reliability on all three measures during the first and second surveys, but poor reliability on the third or final survey.  The Michigan Nurse Educators Sense of Efficacy for Online Teaching scale was found to be reliable on all three applications.

In terms of the actual findings about the PD participants, teachers reported statistically significantly higher self-evaluations of their own teaching, and these evaluations increased over time.  Teachers also reported an increase in their online teaching self-efficacy after each application of the instrument, but the gains were only statistically significant from the first to second and first to third, but not from the second to third (i.e., there was an increase between the mid-point and the final – but not a statistically significant one).

Please note that the presenters had requested that I remove pictures I took of the slides from this session because they did not intend to share the findings outside of those actually present at AERA. To be collegial I have attempted to summarize the content of those slides. If you have further interest in their findings, you may contact them directly because I have shared as much as they have requested of me.

Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Blog at WordPress.com.

%d bloggers like this: