Introduction

Looking at North Carolina Educational Technology (LANCET) leveraged the resources and expertise of its four partners—the North Carolina Department of Public Instruction’s Division of Instructional Technology, SEIR*TEC/SERVE, North Carolina State University and the Friday Institute for Educational Innovation, and the State Educational Technology Directors Association (SETDA)—to study the state’s IMPACT Model, which is the focus of North Carolina’s competitive and formula Enhancing Education Through Technology grants. The study hoped to explore what happens when the essential conditions for the implementation of educational technology are met in heretofore resource-poor schools? Do teachers teach differently? Do students learn more effectively? And above all, does student achievement improve? Must all essential conditions be in place for optimum teaching and learning to take place? Is one essential condition more important than any other?


Over a three-year period, the goal of this model, based on NCLB/EETT goals, was to: 


• Help teachers integrate technology into their teaching practice by providing the technology personnel, resources, and access necessary to implement an Outstanding media and technology program as determined in the Evaluation Rubrics within IMPACT: Guidelines for Media and Technology Programs (http://www.ncwiseowl.org/impact/evaluation.htm#Program%20Evaluation%20Rubrics)

• Provide the necessary personnel, resources, access, professional development, and student instruction to produce technologically literate students by the eighth grade and 

• Have a significant impact on the academic achievement of the school’s student population. 

LANCET’s comprehensive research design that includes experimental, quasi-experimental, and case study components is designed not only to determine which essential conditions of technology make the greatest difference in student achievement, but to build evaluation capacity within the state agency, its school systems, and schools and disseminate instruments, research, and lessons learned to educational leaders across the nation.

Methodology

Project Design

In order to conduct the evaluation of the IMPACT model, the central evaluation team at NCSU and the Friday Institute designed a quasi-experimental (matched subjects) longitudinal evaluation looking at multiple outcomes, which included teacher, student, and administrator skills, attitudes, and behaviors.   Each IMPACT school was carefully matched with a similar school that could serve as a comparison.  Schools were matched based on factors such as geographical proximity, demographics, and school configuration, and multiple measures were used to assess outcomes at each level.  

Within the experimental component of the study, IMPACT schools were randomly assigned to IRCMS treatment or comparison condition, and recruits from geographically proximal schools were also assigned to condition. Multiple measures were used to assess outcomes for teachers and students.



Research Questions:

QuasiExperimental Design


For the quasi-experimental (IMPACT) component of the study, multiple variables were examined through a longitudinal repeated-measures approach.  The hypotheses used took the following form:

Ha:  Students or teachers participating in the IMPACT program will show greater change from the baseline through the third year than matched schools not participating in IMPACT
    μ1d - μ1a  > μ2d - μ2a
Ho:  Students or teachers participating in the IMPACT program will not show greater change from the baseline through the third year than matched schools not participating in IMPACT
μ1d - μ1a  = μ2d - μ2a 

Variables that were addressed within the study included teacher technology use, teachers’ attitudes toward technology, teacher retention, student reading, math, and writing achievement, and student technology skills.



Experimental Design


Within the experimental design component (IRCMS), separate research questions were posed.

Students

1. How changes in reading comprehension varied across condition?

2. How changes in self-efficacy for reading varied across condition?

3. How changes in metacognitive awareness varied across condition?

4. How changes in strategy knowledge varied across condition?

Teachers

1. How changes in self-efficacy for teaching reading comprehension varied across condition?

2. How changes in metacognitive knowledge varied across condition?

3. How did teachers in the two IRCMS conditions vary in terms of treatment integrity?




Results

In all three years of the study, each IMPACT school had a different set of goals and objectives depending on its needs and environment.  While each school made significant progress toward their stated objectives, there was variation in the extent to which each school implemented each component of the IMPACT model.  Compared to Year Two, fewer schools included objectives that directly related to elements of the IMPACT model.  Still, the major components of the model appeared to be in place across project schools by the end of the final year of implementation.  


Overall, results for IMPACT schools, teachers, administrators, and students were generally positive, but with complex patterns occurring related to differential levels of implementation, especially with regard to student achievement.  In some cases, as with teacher attitudes toward technology and stage of concern for the intervention (CBAM), unanticipated patterns occurred, with teachers showing less positive results than expected. We stress that all results from the project should be interpreted with care, and should only be generalized to similar (e.g., Title I) contexts. 

Within the experimental design component of the study, the IMPACT treatment students scored significantly lower than the non-IMPACT treatment and the IMPACT comparison groups.  With regard to the changes in self-efficacy for reading, analyses showed that the non-IMPACT treatment group scored significantly lower than all other groups.  Finally, analysis of self-reported metacognitive strategy awareness revealed that the IMPACT treatment group reported significantly higher awareness than the non-IMPACT treatment group and the IMPACT comparison group.  No significant teacher outcomes were reported.  Again, we stress that these results should be interpreted with care.

North Carolina Department of Public Instruction: http://www.ncwiseowl.org/impact/

North Carolina State University IMPACT Website: http://ced.ncsu.edu/impact/

SERVE Center at UNCG (formerly SEIR*TEC): http://www.serve.org/Evaluation/Capacity/index.php 

IMPACT Website: http://www.ncwiseowl.org/Impact/LANCET/default.htm