This study employed a Delphi design to attain consensus among 38 participants from five groups (high school online practitioners, instructional design academics, university pre-service instructors, online high school administrators and high school online instructional designers) to determine which instructional design competencies from seven organizations (see Table
1) are best suited for online high school teachers who modify their own courses. A Delphi design is a robust method for determining which policies, positions—or, in this case, which instructional design competencies—are considered the most essential (Scheibe et al.
1975).
Table 1
Organizations that have produced instructional design competencies and/or standards
There are multiple ways of conducting research using the Delphi design, including the paper-and-pencil and computer-based method (Linstone and Turoff 1975). The RAND Corporation developed the Delphi design in the 1950s to achieve trustworthy agreement in judgment of experts (Dalkey and Helmer 1963). It is useful specifically for a group communication process (Linstone and Turoff 1975) and for minimizing conflict in studies for which disagreement among participants is potentially high, as when participants come from different professions and backgrounds (Melpignano and Collins 2003). While a high attrition rate and participant access to the Internet and other technology issues are disadvantages (Donohoe et al. 2012), the Delphi design allows participants to make and refine decisions with anonymity (Donohoe and Needham 2009; Edgren 2006), and thus without pressure (Rowe and Wright 1999) from other participants or from the researcher.
To reach a final consensus on competencies for online high school teachers, this study used the Likert-type rating scale, which Scheibe et al. (1975) describe as “quick, easy to comprehend, and psychologically comforting” (p. 267) for participants. Though challenging, choosing participants who sufficiently understand the problem at hand is essential in a Delphi design study (Linstone and Turoff 1975). As well, there is a need to mitigate potential “misunderstanding…from differences in language and logic if participants come from diverse cultural backgrounds” (Linstone and Turoff 1975, p. 7). Thus, this study targeted participants who not only understood the problems facing online high school teachers, but who were willing to work with each other to come up with a solution.
I found the participants by several methods, including conducting a systematic Internet search for online high schools and post-secondary schools that taught pre-service teachers and instructional design in Canada and the U.S. I also used academic membership lists and listservs. I sent over 800 individual questionnaires via email to potential participants.
Participants were drawn from five groups:
-
Practitioners with at least five years of online high school experience, who worked full time in an online high school environment and were directly involved in developing and editing course content.
-
Instructional design academics with a doctorate in instructional design or related field, who were up to date with current instructional design standards.
-
University pre-service instructors who worked with pre-service teachers for at least five years, had online teaching experience, and created online educational content.
-
Online school administrators who had online teaching experience and at least five years of experience in their current administrative positions.
-
Practicing instructional designers who created high school online courses as a whole or in part.
I required participants to have at least five years of experience in their field, though most had over ten years of experience. Eighteen out of 31 participants held doctoral degrees, 11 held master’s degrees, and two had bachelor’s degrees.
The study had three rounds and took 41 days to complete. The participants had ten days in each round to return a completed instrument, which was in PDF form to allow participants to save their answers and come back to them later if needed. A one-week break followed each round to allow time to compile results and create the next instrument. To increase completion rates, I mailed reminder letters (via Postal Service) the day after sending the instrument by email. I also sent a reminder email two days prior and one day after the due date for each round to any participant who had not yet submitted the instrument. I also planned to telephone any participant who had not submitted their instrument three days after it was due.
Over the course of three rounds, participants reduced 116 instructional design competencies, gathered from seven organizations, to 10 competencies that were most applicable for online high school educators who modify their own courses. Participants in this study acted as one whole group to move a competency forward. Thus, during the first round, competencies ranked as ‘Strongly Agree’ by 60% of participants or more indicated agreement among participants in the first round. A consensus of 60% was used in the first round to make sure none of the items would be left out. The remaining two rounds used a 75% ranking ‘Strongly Agree’ to indicate consensus on an item. While the use of percentage has been commonly used to indicate agreement among participants, there is not an agreed upon definition of what constitutes consensus in a Delphi study (Hasson et al. 2000). This study followed the majority of studies in using a 75% ranking ‘Strongly Agree’ to indicate consensus on an item (Johnston et al. 2014; Penciner et al. 2011; York and Ertmer 2011).
To complete each instrument, the participants were asked to use their professional judgement to evaluate a list of competencies using a Likert scale to indicate whether the online high school teachers required the listed competency to alter course content, assignments, LO and other elements:
-
Strongly agree – The competency should be required;
-
Agree – It is a suggested competency, but not required;
-
Neutral – Do not have an opinion;
-
Disagree – Probably should not be a competency;
-
Strongly disagree – The competency should not be required
The most important item on the Likert scale for this study was ‘Strongly Agree’. This ranking determined whether a competency moved to the next round.
Table 1 describes the different organizations that have created competencies and/or standards. Only instructional design competencies from these organizations were used in this study.
Subscribe: Feedly
12 Unique Blogs Are Written By Professors
