The Tennessee EdTech Accountability Model (TEAM) is a rigorous, scientifically based high quality evaluation of those interventions. Three research questions were formulated to help guide the study. The research questions asked the following:
1. Does implementation of the TnETL model raise student achievement?
2. Does implementation of the TnETL model increase the use of research-based classroom practices?
3. Does implementation of the TnETL model improve students’ ability to use technology as a tool?
To answer the research questions, comparative analyses were conducted on classroom practices, teacher attitudes, student performance outcomes, and student achievement. Descriptive analyses were used to describe reactions to TnETL Program by teachers, principals, and technology coaches. The evaluation methodologies involved using the Formative Evaluation Process for School Improvement – Technology Package (FEPSI/TP). Data gathered through the FEPSI/TP were derived from direct classroom observation, teacher surveys, interviews/focus groups, technology implementation benchmarks, and analysis of student achievement.
The data gathered from direct classroom observations were derived through whole school and targeted observations. In this sector of data collection, challenges were faced in areas of implementation, but were overcome with changes to planning, scheduling, and communication methods. The implications for replication were that the instruments used were valid and reliable, national norms are available for comparisons, online submission was available and at a reduced cost over paper-based versions, continued use reveals change over time, and descriptive results were available online to schools, districts, program participants and the state. There were concerns with this method. This method of data collection requires trained observers; school based for formative evaluation and external for unbiased formative and summative evaluation. After the data collection has occurred, researchers are needed to conduct inferential analyses and generate useful reports.
Teacher surveys were also used to collect data. Challenges were noted along the way, as well as how they were overcome. Again, scheduling, paper vs. online surveys, and the availability of off-site observers for faculty meeting were some of the more prominent challenges. These problems were overcome again through better planning, paper surveys, and additional back-up observers.
The method of data collection that involved interviews and focus groups included principal interviews, teacher focus groups, and technology coach survey and interview. Concerns with this method were scheduling, accurate teacher lists, scheduling time and location, and absenteeism on the day of the focus group. Working with the Coach to schedule meetings, including principal interviews with other data collection strategies, using a teacher list for targeted visits, and randomly selecting more teachers than needed helped to alleviate the challenges in this particular category.
The technology implementation benchmarks presented their own set of challenges. Teacher and Principal buy-in, training teachers and principals to develop and effectively use benchmarks, and the timely use of the instrument were perceived to be the biggest challenges. These challenges were overcome by providing information packets, face-to-face training, en expert reviews to build rationale and trust in using the benchmarks; provision of a benchmark template, example templates, hands-on training, and expert review of each school’s initial benchmarks; and regular communication with the Technology Coach.
The student performance-based assessment portion noted challenges in the area of availability of schools that met the criteria, (8th grade students, computers, and teachers with basic computer skills); obtaining principal permission, teachers “waiting till the last minute” or not administering the assessment. These issues were resolved by using easy to understand solicitation materials, using easy to administer teacher-proof materials, adjusting the number of participants, and frequent communication.
The last data category in the FEPSI/TP was student achievement. The major challenges to obtaining this data were obtaining student pre-intervention and yearly performance data from the state due to lack of knowledge of who to contact and the ability to verify approval to receive that data in a timely fashion. The method by which these challenges were overcome was to make the request early and include all critical information that was needed in the request.
Plans for dissemination and presentation, as well as lessons learned about conditions for replication necessarily have been considered. Various ways and places have been considered as outlet methods. School level reports, state level reports, SETDA website, CREP website, articles and presentations were just some of the suggested methods for dissemination of the data. Lessons learned about replication such as building evaluation accountability into the competitive grant, collecting common data from all grantees, and having a high level of accountability from the state are all subject areas to be broadcast and shared for similar studies.
