Introduction
Wisconsin’s Evaluating States Educational Technology Programs research project “A Wisconsin Proposal to Study the Effectiveness of Two Models of Implementing Educational Technology” was implemented for the purpose of evaluating the impact of using technology within two constructivist models when implemented by teachers teaching seventh and eighth grade science and social studies. The two models selected were the Big6™ Information Literacy and 6 + 1™ Traits Writing. The models selected were considered the best from an RFP released to all former Wisconsin Title II, Part-D competitive grant recipients. Guidelines included the requirement that proposal designs included the ability to measure student progress over time and that interventions could be replicated in other school districts. The results of interventions by the experimental groups were compared to the results from a control group comprised of a similar cohort of students representing the demographic make-up of the State of Wisconsin. Data collected and analyzed included the following:
1. Two teacher questionnaires administers several times each year
2. Student questionnaire administered fall/spring of each year
3. Achievement data
4. Classroom observations
5. Student work sample analysis for higher order thinking skills
The Wisconsin Department of Public Instruction in partnership with the University of Wisconsin-Milwaukee was interested in determining the impact of using technology with the two chosen experimental models versus the control group. School districts were selected based on a specific application process and were randomly assigned to one of the experimental groups or the control group.
Statement of the Challenge
Active student engagement in learning has been a concern in Wisconsin. Despite comparatively high national test scores, the State of Wisconsin continues to have a wide achievement gap between minority and white student populations as well as a significant gap between students of low income and those of average and above average family incomes.
Recognizing our students are digital learners who use multiple tools for amusement and accessing information including computers, cell phones, video games, movies etc, the DPI wanted to research the impact of a more problem based learning approach to teaching and learning through the use of technology. The challenge faced by the department was to quantify the impact of combining technology with constructivist models in the areas of social studies and science.
Research Questions
We were interested in several factors related to the design and implementation including the following:
1. What affects would the models have on student engagement?
2. What affects would the models have on developing higher order thinking and meta-cognitive skills as measured by student work samples?
3. Would significant, in-depth and ongoing professional development change teacher expectations and instructional behaviors that would result in improved student engagement, student achievement, and development of higher order thinking skills?
4. Would the models and supportive training be replicable in other school environments?
5. If resulting data did not show statistically significant improvements, were there lessons learned that could be applied to future reform efforts in the State of Wisconsin?
Methodology
Participants
This study was a longitudinal analysis of students as they progressed from the beginning of seventh grade in the 2004 – 2005 academic year through their completion of eighth grade in 2005 – 2006. Subjects were selected from a statewide sample of 34 middle schools and school participation in the study was voluntary. Using the U.S. Census Bureau’s criteria for identifying urban, rural, and suburban a total of 35 schools were selected with 16 defined as suburban, 7 as urban, and 12 as rural. The sample of 34 middle schools consisted of approximately 70 7th grade teachers and 3700 7th grade students assessed throughout the 2004-2005 academic years. The 2005 – 2006 academic years included a sample of 34 middle schools with approximately 70 8th grade teachers and 3500 7th grade students. The number of students participating in the 2004 – 2005 and the 2005 – 2006 academic years was approximately 1600.
Teacher Training
In the summer of 2004, teachers from the participating school districts assigned to the 6+1 Trait Writing® or the Big6TM treatment groups were required to attend training sessions on their assigned instructional models. Teachers assigned to the control group attended workshops unrelated to the treatment models. The sessions were conducted by consultants who specialized in training teachers on the respective models. Treatment group teachers, were administered a pre-test questionnaire at the beginning of the first day of training in an attempt to measure existing knowledge of the treatment models and to ensure congruence of treatment model familiarity between groups. At the conclusion of the training workshop, a posttest teacher questionnaire was administered to only the teachers assigned to the 6+1 Trait Writing® and the Big6TM groups to determine the extent to which the workshops increased teachers’ knowledge of the treatment models. Both the pre- and posttest questionnaires asked teachers on a five-point Likert scale how comfortable they felt (i.e., not comfortable to comfortable) and their capability incorporating tenets of the 6+1 Trait Writing® and the Big6TM into their daily instructional practice, as well as how comfortable they felt integrating technology within the instruction of the respective model. Thus, the overall purpose of the teacher training session assessments were to determine the efficacy of the training sessions by considering how teachers would: (1) implement the training model within their classroom instruction and (2) integrate technology in the instruction of the training model. Specifically, the goals for administering the pre- and posttests were to examine whether differences occurred between the control, 6+1 Trait Writing® and Big6TM groups in their pre-existing knowledge of the models as well as to measure change that may have occurred in teachers’ ratings of their capability in teaching the learned skills following the training sessions.
In addition to the pre-test assessing teachers’ familiarity with 6+1 Trait Writing® and Big6TM models, it also included questions that asked teachers if they had attended workshops or trainings over the past five years on higher-order/critical thinking instruction. The purpose of this additional questioning was to assess the level of complexity teachers require of their middle school students in classroom work.
The formal training on the two experimental models (Big6 and 6 + 1 Traits Writing) consisted of an initial 40 hours during the summer of 2004.
The focus of the training was:
1. Model design and implementation both to incorporate inquiry based teaching and learning.
a. Big6 schools also had a local “coach” trained to support their techers.
2. Technology integration training
3. Apply model to one instructional curricular unit in preparation for implementation
a. Expectation was that 6 Traits instructors would incorporate 6 + 1 Traits Writing as frequently as they could in all lessons.
b. The Big6 instructors were required to deliver 2 instructional units throughout the school year
There were two additional training dates during the school year, one in October and one in February of each of the two years of intervention. This training was to share and learn with each other based on their experiences as well as to refine some of their implementation skills. An additional 40 hours of training was held for teachers during the summer of 2005. For additional preparation time, teachers were also allowed up to seven days of release and collaboration time throughout each school year.
In addition to the face to face training, each of the project facilitators/trainers, held online support sessions with specific expectations of instructors in both experimental groups. For example, each teacher was required to submit an outline of their instructional objectives, how they were going to implement specific aspects of the model and to provide feedback on the results of the instruction and student learning. This online forum was very helpful for many teachers, however, there were approximately 20% who had limited involvement in the online support.
Scores for both the pre- and posttests were standardized in order to compare across groups, since group sizes were unequal. A multivariate analyses of variance (MANOVA) was conducted, with treatment group as the independent variable (Big6TM, 6+1 Trait Writing®, Control) and four dependent variables (including 6+1 Trait Writing® familiarity, 6+1 Trait Writing® and technology integration, Big6TM familiarity, and Big6TM and technology integration) to determine whether all groups, regardless of treatment assignment, entered the training sessions with equivalent knowledge about the models. Results indicated that all groups were equal in there model familiarity, with and without integrating technology (F =1.23, p=.29, α=.05).
Four paired-samples t-tests were conducted to determine whether the training workshops led to an increase in how teachers’ self-rated their own teaching ability in: (1) the treatment models (Big6TM or 6+1 Trait Writing®), and (2) the integration of technology within the instruction of models (refer to Table 3 for Descriptive Statistics for Workshop Assessments).
Using the treatment models scores, the t-test analyses revealed that teachers in the 6+1 Trait Writing® group rated their abilities significantly higher (t = 6.04, p < 0.01) following the workshop training when compared to their perceived abilities prior to the workshop. Teachers in the 6+1 Trait Writing® group also indicated significantly higher (t = 2.79, p = 0.01) abilities following the training in the integration of technology with the tenets of the 6+1 Trait Writing® model. The teachers who attended the Big6TM training also demonstrated significant differences in pre- versus posttest scores (t = 6.32, p < 0.01) and in rating their capability in integrating technology with the models (t = 5.37, p < 0.01). Overall, the results indicated that teachers from both treatment groups had significantly higher posttest self ratings of their perceived ability to implement the treatment models and to integrate technology within the instruction of those models. Thus, it was concluded that the workshop improved teachers’ perception of their ability to implement tenets of the Big6TM or 6+1 Trait Writing® models and to integrate their respective model with technology.
Results
Model Fidelity
In general, teachers incorporated their respective instructional model (e.g., 6+1 Trait Writing® or Big6TM) approximately 1 – 3 times a month and when they did engage the students teachers used a little less than half the class time dedicated to teaching their respective model. Overall, model use and engagement was moderate with no practical group differences. Furthermore, the 6+1 Trait Writing® remained fairly consistent in spending approximately half the class time engaging students in activities related to the 6+1 Trait Writing® model. The Big6TM teachers remained relatively consistent as well in spending about half the class time engaging students in activities related to the Big6TM model.
Students felt that they rarely learned about their respective instructional model (e.g., 6+1 Trait Writing® or Big6TM). This was slightly lower than teachers’ reported instructional engagement of the models with their students. Though there were slight differences between groups, these mean differences did not exhibit a practically sizable effect. In other words, over the course of the study mean differences between groups were small and not practically meaningful.
Overall, teachers reported teaching their respective treatment model to their classes somewhat more than what students reported learning about the model. Observational results corroborate with teacher and student reports. However, despite similar results across data sources, the models were not taught on a regular basis, which yield concerns for the validity of students’ skills in application of the model as measured by their school achievement.
Technology
In general, the teachers indicated moderate levels of computer and technology use with some significant differences over time and between the control, 6+1 Trait Writing®, and Big6TM groups. Overall, the 6+1 Trait Writing® group used technology slightly more than the other two groups consistently over time. Teachers had students using computers to complete assignments every few weeks or at least once a month with the 6+1 Trait Writing® group showing a slight increase in computer use. In general, students rarely used software with the exception of the 6+1 Trait Writing® group showing a significant increase in software use. Finally, teachers reported believing they possessed average skills in their use of computers and the internet. Teachers’ perceptions of their computer skills were relatively consistent across time regardless of their assigned treatment group.
Student responses on the questionnaire were similar to those expressed by their teachers. In general, students reported infrequent use of technology and computers in school. A closer examination of students’ responses on the questionnaire by their treatment group revealed significant differences between groups, however the effects of the mean differences between groups were small. The 6+1 Trait Writing® students consistently felt they used computers and technology more than the control and Big6TM groups. Thus, the control and Big6TM groups used technology about the same amount of the time. Though, again the group mean differences were generally small.
In terms of student achievement data from standardized tests, whether or not schools used technology accounted for a small amount of the differences in science achievement scores across schools at the onset of the study, but growth in these same scores was not due to technology use. For social studies achievement, the use of technology accounted for a significant portion of the differences across schools at the onset of the study, but over the two years social studies achievement growth was not accounted for by either technology or any additional predictors.
Overall, teacher and student questionnaire results corroborate technology use on a biweekly basis. However the 6+1 Trait Writing® used technology slightly more than the other two groups consistently over time.
Constructivism
In general, teachers did not have high expectations of engaging their students in learning by constructivist type teaching practices. Though, the 6+1 Trait Writing® group had higher expectations of constructivist learning than their Big6TM counterparts. However, these expectations were not greater than the control group for the 6+1 Trait Writing® group. Further, teachers rarely used constructivist type activities within their classroom and this was consistent across instructional groups. Thus, while teachers across groups slightly agreed with constructivist beliefs, there were no significant changes in teachers’ reported beliefs from the beginning of 2004, to the end of May 2006. Additionally, teachers did not differ significantly in their constructivist beliefs regardless of the treatment group to which they were assigned.
Overall, the integration of technology and constructivist instructional activities in the classroom remained relatively stable. The Big6TM teachers did report a slight increase in constructivist activities, but this was not supported by the students who reported no significant increases in constructivist activities. In general, teachers reported using constructivist activities 1 to 3 times a month and students reported constructivist activities sometimes being integrated in the classroom. Further, classroom observations revealed that roughly half the classroom and instructional time in all groups were based in the tenets of constructivist practices; yet no significant changes occurred from the beginning of 2004, to the end of May 2006.
Contacts
Director, Instructional Media and Technology Team
Wisconsin Dept. of Public Instruction, PO Box 7841
Madison, WI, 53707-7841, USA
stephen.sanders@dpi.state.wi.us
Business: (608) 266-3856
Business Fax: (608) 266-2529
