Program Recommendations

• The successes observed in this study provide additional reason to continue to expand the program. However, because participating schools were self-selected, it should not be inferred that the program should be encouraged for any school who does not really want it.

• While the state may not want to dictate program recruitment and selection policies, the importance of more proactively reaching out to populations who are less likely to seek out the program, especially traditionally underserved or under-represented groups, might be further stressed as programs mature, with an emphasis that casting a wide net is not sufficient to fully ensure diversity.

• The observation rating form could serve as a useful tool for focusing discussions around “site health” visits.

• Further study of program impacts could be valuable for helping to identify whether the skills and attitude changes that EAST develops have long-term impacts on students’ further education, or even on choice of career paths.

• Further study may also be warranted in order to more closely examine the extent to which program benefits may be influenced by other affective student characteristics, and the extent to which the program may be able to help students overcome certain characteristics such as a lack of interest in school. This could help further strengthen the value of the program by achieving a more targeted diversity, identifying and encouraging participation from types of students who might be shown to benefit but who traditionally might not apply. Since there are likely to be a number of student characteristics that are relevant to success other than the demographic indicators available from school records, school personnel who know the students may be in the best position to identify appropriate candidates. The targeted recruitment efforts that were often used by the schools can therefore be an effective way to accomplish this balance, as long as referring staff use appropriate criteria. We agree that diversity is a laudable goal for EAST enrollments, but it is not clear whether this should necessarily mean that enrollment should be representative of the school in all cases.

• Another potential objective for further research would be to explore the impacts of better established programs. It is quite possible that programs that have had the opportunity to become fully established might demonstrate even stronger impacts for a larger number of outcomes.

Evaluation Methodology Recommendations

• The value of development of a logic model early in the study can not be overemphasized. Particularly for a program such as EAST, for which adaptability and responsiveness to specific circumstances are among its most important distinguishing features, it is critical to clearly define the parameters within which this flexibility is meant to exist, both to help new practitioners understand what is expected of them, and to help researchers recognize the program that is being evaluated. This process can be quite time consuming, particularly if the program design has never been explicitly laid out before, but it is in this very situation when it is needed the most. It should be noted, however, that it may not be possible to create a written program description that is detailed enough for someone with no prior familiarity to implement or to evaluate the program. It may very well be that a program such as EAST requires personal training from existing staff to be replicated or evaluated, which would have significant implications for the cost of replication.

• Measuring program fidelity is a crucial part of program evaluation, especially when the programs that are being studied are in the earlier stages of implementation. Observation is an important method for collecting evidence of fidelity, especially when interactions between students and teachers form an essential part of the model. However, there may also be critical aspects of program fidelity that cannot be assessed through passive observation alone, especially if resources do not allow for frequent enough observations to obtain a representative picture of the program in a short period of time. In such cases, interviews and/or surveys of site-based program staff and of monitoring staff can provide an invaluable complement to observation, and can also access potentially important factors (such as teachers’ attitudes and beliefs) that would be extremely difficult if not impossible to infer from observation.

• Assessing the possibility of the existence of selection effects can be exceedingly difficult. It may be worth implementing additional behavioral or attitudinal measures to try to evaluate student characteristics that are expected to influence program outcomes, even if they do not represent goals of the program per se. However, since it is presumably impossible to directly assess all potentially salient variables, it is equally important to make inferences about the likelihood of unmeasured selection effects by examining the procedures used to identify participants. 

• Conducting a randomized evaluation design in a public school setting is extremely challenging, and in some circumstances, its pitfalls can be so severe that it may not be the best approach. Where the conditions exist that make it possible—most notably, the opportunity to randomly select schools (or classrooms) for participation, or at least to randomly assign applicants to participating and control conditions—there are a number of additional conditions that must be achieved in order to maintain the integrity of the study. One of the most important issues is to ensure the cooperation of participating schools—including the control schools—throughout the study. Possible strategies can include providing incentives, involving the schools in discussions about the value of the evaluation and in the process of designing the evaluation, providing schools with formative feedback throughout the study to make it more valuable, sanctioning of the evaluation from authorities, and laying out the obligations of participation in the evaluation as a condition for program participation. Nevertheless, it is important to avoid making schools feel like the study is being forced down their throats, as lack of cooperation can invalidate the randomization process. This is most challenging when control schools are being asked to participate actively in the study. Unless the program is already considered desirable and can be offered as a reward for cooperation, it might be best to design the study in such a way that only the most minimal involvement is required from control schools.