The research design for the third and final year of the ED PACE project emerged from 1) a review the Year 2 findings, especially those that emerged from linkage analyses and allowed us to pinpoint some key factors related to facilitation of learning and student outcomes; 2) a statewide meeting with lead teachers, adjuncts, facilitators, convened to share two years of data and explore ways to combine what the research had shown and what practitioners had learned to enhance the Virtual School Spanish Program; and 3) a debriefing with WVDE partners.

The focus of the Year 3 design was facilitation, or the factors that, in concert, support Spanish learning in the Virtual School Spanish program. Continuing research conducted in Year 2, we also included plans for further examination of the links between facilitation or classroom implementation and student performance. In the Year 3 design, we proposed to:

• Describe how facilitation of Spanish learning occurs in the blended Virtual Spanish instructional model, focusing on factors such as the use of technology, and the variation in use and type; high-quality instruction, including activities that encouraged students to make connections to the real world, group or dialogue activities, combination of all language-building activities—listening, speaking, hearing, and reading; adult language use; feedback and communication; and cultural connections, community-building activities, and other elements of beyond those solely related to teaching language skills.
• Describe supports to the facilitation of Spanish learning in the Virtual Spanish instructional model, including collaboration and communication among team members, shared roles and responsibilities, instructional flexibility, and professional development.
• Examine linkages among supports to facilitation and the implementation of factors that directly facilitate Spanish learning.
• Examine linkages between the implementation of factors that directly facilitate Spanish learning and student intermediate and learning outcomes.
• Examine the linkages among student intermediate outcomes and student learning outcomes.
• Examine the degree to which students are prepared to meet the expectations of Spanish II.
• To address these objectives, we further refined instruments, streamlining the observation protocol; revising the facilitator and student online surveys; creating new protocols for adjunct, lead teacher, and facilitator interviews, student focus groups, and high school Spanish II teachers; and revising the Spanish Assessment based on review of expert panel and item statistics.

Because only eighth-grade 1B students take the Spanish Assessment, we observed only 1B classes in Year 3 (N=17). To get a view of individual classes more detailed and nuanced than that afforded by a single visit, we observed each 1B class for two consecutive days, and in a few cases made a return visit, for a total of 56 observations across the 17 1B sites. During visits, we conducted post-observation interviews with facilitators (N=23) using a structured protocol, and informal focus groups (N=25) with five to six students in most 1B classrooms. Where possible, we talked informally with school principals. In Year 3 we also conducted in-person interviews with 12 Virtual Spanish adjunct teachers.

To extend our picture of the impact of the program on students’ language skills, preparation, and attitudes toward foreign language, we interviewed Spanish II teachers in a sample of high schools (N=10) that offer the next level of Spanish instruction to Virtual School students, and four program leaders and administrators. During interviews with facilitators, adjunct and lead teachers, and program leaders, the focus of the questions was on the facilitation process and implementation factors such as classroom activities, collaboration, communication, student feedback, and program support.
In five sites identified in the first round of visits as examples of effective facilitation, we made an additional one-day visit. Those sites were chosen based on a set of criteria that included a particular style of facilitation, history and stability of the program, geographical representation, and past student Spanish assessment results. To have as full a data set as possible, we included sites that were also SOPA sites.

Data Analyses

Analyses of the impact and implementation data fall into four categories: 1) descriptive analyses of all items on the student, parent, facilitator, and adjunct surveys and of observation indicators and other open-ended, qualitative data to identify preliminary findings and themes; 2) psychometric analyses of the different measures associated with the various constructs listed in Table 1 above to determine their reliability and validity as measures of these constructs; 3) examination of the variation in student academic and non-academic outcomes, and implementation outcomes and processes, to identify factors that may differentiate sites, or students within sites, from each other; and 4) use of statistical models to assess the strength of relationships among implementation processes and outcomes, intermediate academic and non-academic outcomes, and learning outcomes.  

Quantitative Analyses

We conducted several analyses to summarize survey and classroom observation data, examine differences and trends in Spanish assessment performance, and explore relationships between implementation characteristics and student performance and with classroom facilitation. These analyses helped us answer key research questions about the nature of the program, student outcomes (e.g., Spanish proficiency, valuing of foreign language, engagement, and intent to continue to study foreign language), and about relationships between constructs and student and instructional outcomes.  

We used descriptive statistics (means, standard deviations, and frequencies) to summarize key survey questions. We created scales from survey items and from observation measures by calculating an average across all items or measures for each scale. In addition, we examined the reliability and item statistics for each of the scales . To assess differences in Spanish achievement between virtual Spanish and face-to-face students, we conducted analyses of covariance, adjusting Spanish achievement scores for prior year Language Arts achievement scores and adjusting for the effects of classrooms . We also calculated appropriate effect sizes (mean differences and eta-squared or proportion of variance statistics) to determine the practical significance of any findings. To examine trends across the three years of the study for the virtual Spanish students, we conducted repeated measures analysis of covariance, adjusting Spanish achievement scores for prior-year Language Arts achievement scores, and adjusting for the effects of classrooms and calculating the appropriate effect sizes.

To assess the links between implementation indicators and student outcomes, we conducted a set of correlational analyses where we predicted the level of Spanish achievement on each of the three multiple choice and four writing performance measures by levels of the different characteristics in the sets of variables . We used hierarchical linear modeling regression techniques to analyze the data because students are nested within sites. Because of this nesting effect, students in a single site are more likely to be similar to each other than to students in other sites since they share a common context. The hierarchical approach allowed us to account for the similarities in students’ performance that are due simply to their being in the same context, and to correct the significance test results to obtain better estimates.  

Qualitative Analyses

To address a number of research questions related to implementation, we gathered and analyzed qualitative data during the project’s three years, through open-ended survey questions and interviews with key participants and stakeholders. For open-ended survey responses, we generated a series of codes based on recurrent themes that emerged from multiple reviews of responses, then coded all responses. The interview and focus group data were digitally recorded and uploaded to a computer. Using a qualitative data analysis software program, we then coded and analyzed the data. For the first level of data analysis, we coded the data according to the constructs addressed in the questions and aggregated by type of respondent. In the second level, we aggregated the data by construct. In analyzing the survey and interview data, we looked for consistencies and common patterns across responses.