Welcome to SETDA's National Leadership Institute's Toolkit 2006. This site was created as part of the National Leadership Institute in 2005 in Washington, DC.
   
 |
|
Supporting Research: Statewide Data SystemsStudent Data Matters and Here’s Why! There is a growing body of research about the value of data-guided decision making in the educational arena. Within the accountability context established by the No Child Left Behind Act, it is essential that educators have the information they need to make informed decisions – from the classroom practice to boardroom policy. While this approach - often called D3M, for data-driven decision making – has its roots within the business world, it has been increasingly touted in the educational literature over the past decade or so as educators try to implement structures and processes that will improve educational effectiveness.
To start with, we must be able to quantify student learning in meaningful ways – for teachers, for administrators, for legislatures and government agencies, for private grantors and funding foundations, for parents, for the broader community, and, of course, for the students themselves.
Then, we need to be able to use this information about student learning to make decisions about the next educational steps for each student, and about the programs and practices within our schools that will best meet students’ needs. For students who are struggling to reach the standards, we need to be able to hone in on their specific learning needs and target teaching accordingly. For students who are successfully achieving the standards, we need to be able to expand and enrich their knowledge and skills. The first four references in the Annotated Bibliography provide discussion on some of these critical questions.
After we have collected, analyzed, and acted upon this critical information about student learning in our classrooms and schools, we need to be able to tell our communities about the return they are getting for their investment in education.
The other four references in the Annotated Bibliography discuss aspects of system-level data analysis and reporting.
Following are synopses of these selected references that clearly support the need for and value of student-level data in directly improving academic achievement as well as determining the effectiveness of our investments in educational practices. Black, P. & Wiliam, D. (1998). Inside the black box: Raising standards through classroom assessment. Phi Delta Kappan, 80 (2), 139-148. Black, P., Harrison, C., Lee, C., Marshall, B., & Wiliam, D. (2004). Working inside the black box: Assessment for learning in the classroom. Phi Delta Kappan, 86 (1), 9-21. Paul Black and Dylan Wiliam conducted a meta-analysis of studies on academic innovations and identified more than 40 showing that strategies that include strengthening the practice of formative assessment in the classroom produce significant and often substantial learning gains. They use the general term assessment to refer to all the activities undertaken by teachers – and by their students in assessing themselves – that provide information to be used as feedback to modify teaching and learning activities. Such assessment becomes formative assessment when the evidence is actually used to adapt the teaching to meet student needs.” Typical effect sizes of the formative assessment experiments were between 0.4 and 0.7 – larger than most of those found for educational interventions. Gains of this magnitude represent significant learning – an effect size of 0.4, for example, would mean that the average pupil involved in a particular learning experience would record the same achievement as a pupil in the top 35% of those not so involved. An effect size gain of 0.7 in the recent international comparative studies in mathematics would have raised the score of a nation in the middle of the pack of 41 countries (e.g., the US) to one of the top five. Thus it is critical for teachers to know what students are learning and what they have not, and use that “data” to make the next instructional decision for each student. Black and Wiliam promote school- and classroom-based teacher professional development to build teacher skills and school cultures around formative assessment. Paul Black is a professor emeritus from the Department of Education and Professional Studies at King’s College in London. Dylan Wiliam is the director of the Learning and Teaching Research Center at the Educational Testing Service in Princeton, New Jersey. Both have worked extensively with teachers, in England and the United States, around classroom-based assessment. Popham, J. (2003). The seductive allure of data. Educational Leadership, 60 (5), 48-51. James Popham, from UCLA, states that the most important data in the United States these days are test data – particularly data describing student performance on achievement tests. But he argues that only instructionally beneficial data – from instructionally useful tests – have real value for improving teaching and learning. Instructionally useful tests are those that measure student attainment of a worthwhile curricular aim; measure something teachable; are based on clear descriptions of the skills and knowledge being measured; yield specific results that can inform teachers about the effectiveness of their instruction; and don’t take too long to administer. He argues that, if we have instructionally useful tests that provide “per-standard” reporting of results to teachers, then teachers have powerful tools to use to focus instruction and improve student learning of standards. James Popham is a professor emeritus at the University of California, Los Angeles. He has written and spoken extensively on the topic of educational assessment. He is considered a leading expert in the field of instructionally sensitive/instructionally beneficial assessment. Marzano, R. J. (2003). Using data: Two wrongs and a right. Educational Leadership, 60 (5), 56-60. Marzano, R. J. (2003). What works in schools: Translating research into action. Alexandria, VA: ASCD. Robert Marzano, from the Mid-continent Research for Education and Learning (McREL), comments that, “schools that use data to make decisions are following some of the best advice from both the world of business and the world of education.” He cautions, however, that schools and districts frequently make two key errors in their efforts to be data driven. First, they often use indirect measures of learning – that is, measures that are not sensitive to the actual teaching and learning occurring in the classroom. Second, they often have no plan or system for interpreting and using the data – that is, there is no accompanying explanation of how to improve. This article identifies 11 school, teacher, and student factors that are the primary determinants of student achievement; and thus, those on which we need to collect and analyze data. The research and rationale behind these factors are provided in his book on What Works in Schools: Translating Research into Action. Robert J. Marzano is a senior scholar at Mid-continent Research for Education and Learning (McRel) in Aurora, Colorado and an Associate Professor at Cardinal Stritch University in Milwaukee, Wisconsin. He has conducted extensive study in the area of “what works in schools and classrooms.” He has developed programs and practices used in K-12 schools and classrooms that translate current research and theory on cognition into instructional and leadership methodology. American Association of School Administrators. (2002). Using data to improve schools: What’s working. Washington DC: Office of Educational Research and Improvement. This report is available from the American Association of School Administrators. With advances in technology and the increased demand for assessing student learning, an unprecedented amount of data is available to educators. In the Foreword to this guide, Paul Houston, Executive Director of AASA, writes, “as educators shift their focus from simply reporting test results to using the data to improve instruction, data become essential ingredients in school improvement. Educators know that the effective use of data can measure student progress, evaluate program and instructional effectiveness, guide curriculum development and resource allocation, promote accountability and, most importantly, ensure that every child learns.” This document was prepared as an easy-to-read guide to using data to drive school improvement. It provides strategies for building school and district cultures of inquiry, and also describes challenges and successes of educators from a variety of districts. This report was produced by the American Association of School Administrators with funds from the Office of Educational Research and Improvement. A team of superintendents from across the United States contributed to development of the report. AASA, founded in 1865, is the professional organization for educational leaders across the US and in other countries. Its mission is to support and develop effective school system leaders who are dedicated to the highest quality public education for all children. Raudebush, S. (2004). Schooling, statistics, and poverty: Can we measure school improvement? William H. Angoff Memorial Lecture. Princeton: Educational Testing Service. This report is available for download or purchase from the Educational Testing Service. Raudebush discussed two way of using available test data to judge school effectiveness and improvement – snapshots of average proficiency and value-added systems. He noted that “recent events have revealed the dependence of our financial system on a flow of accurate information to corporate stockholders. Accuracy of the data flowing from school accountability systems is no less essential to sustain current strategies for educational improvement.” He addressed a four-part investigation:
He concludes that, “when high-stakes decisions are based on statistical evidence, it is sensible to scrutinize the quality of the evidence with great care. Holding educators accountable for their contributions to student learning is a laudable goal and one potentially powerful lever for school improvement. But the amount and quality of data must be reasonably aligned with the uses of data in decision making if the accountability initiative is to earn lasting credibility.” Stephen Raudebush is a professor of education and statistics, and a senior research scientist for the Institute for Social Research at the University of Michigan. He has made an impressive career of bringing advanced evaluative methods to issues of great social import, studying teaching quality, school effectiveness, child development, marital relationships, and criminal behavior. The William H. Angoff Memorial Lecture Series was established by ETS in 1994 to honor the life and work of Bill Angoff. For more than 50 years, he made major contributions to educational and psychological measurement. In line with his interests, this lecture series is devoted to relatively nontechnical discussions of important public interest issues related to educational measurement. Raudebush, S. (2004). Schooling, statistics, and poverty: Can we measure school improvement? William H. Angoff Memorial Lecture. Princeton: Educational Testing Service. This report is available for download or purchase from the Educational Testing Service. Raudebush discussed two way of using available test data to judge school effectiveness and improvement – snapshots of average proficiency and value-added systems. He noted that “recent events have revealed the dependence of our financial system on a flow of accurate information to corporate stockholders. Accuracy of the data flowing from school accountability systems is no less essential to sustain current strategies for educational improvement.” He addressed a four-part investigation:
He concludes that, “when high-stakes decisions are based on statistical evidence, it is sensible to scrutinize the quality of the evidence with great care. Holding educators accountable for their contributions to student learning is a laudable goal and one potentially powerful lever for school improvement. But the amount and quality of data must be reasonably aligned with the uses of data in decision making if the accountability initiative is to earn lasting credibility.” Stephen Raudebush is a professor of education and statistics, and a senior research scientist for the Institute for Social Research at the University of Michigan. He has made an impressive career of bringing advanced evaluative methods to issues of great social import, studying teaching quality, school effectiveness, child development, marital relationships, and criminal behavior. The William H. Angoff Memorial Lecture Series was established by ETS in 1994 to honor the life and work of Bill Angoff. For more than 50 years, he made major contributions to educational and psychological measurement. In line with his interests, this lecture series is devoted to relatively nontechnical discussions of important public interest issues related to educational measurement. Coley, R. J. (2003). Growth in school revisited: Achievement gains from the fourth to the eighth grade. ETS Policy Information Report. Princeton: Educational Testing Service. This report is available for download or purchase from the Educational Testing Service. In this analysis, Richard Coley used data from the National Assessment of Educational Progress to track the performance of students from varying demographic groups and different states to concurrently examine the achievement gap as well as the academic attainment of these groups over time. Specifically, he compared NAEP Reading and Mathematics results from the Grade 4 cohort tested in 1996 to the Grade 8 cohort tested in 2000. He found that the growth in average scores between the fourth and eighth grades is about the same for all student subgroups except Black students; that subgroup gained more scale points than did White and Asian, with the difference equivalent to roughly one year in school. He also found that, while students attending nonpublic schools score higher, on average, then those in public schools, both groups add about the same value between the fourth and eights grade. He believes that “the view of achievement provided by looking at cohort growth gets us closer to measuring what really happens in school.” Richard Coley worked as an Education Policy Analyst for the Policy Information Center at the Educational Testing Services in Princeton, New Jersey. He has conducted research in a variety of areas within the educational arena, including student achievement gains, indicators of school readiness, teacher preparation, and the status of technology in US schools. Grissmer, D., Flanagan, A., Kawata, J., & Williamson, S. (2000). Improving student achievement: What state NAEP test scores tell us. Santa Monica, CA: RAND. This report is available for purchase or download from Rand Corporation. This study used state-level NAEP data to estimate score gains nationally and by state, the effects of varying levels and uses of per-pupil expenditures on student achievement, and the cost-effectiveness of the major alternatives for utilizing educational resources. The report noted that, “overall, the results paint a more positive picture of American public education than is commonly portrayed, especially with respect to the effective allocation of resources.” Both the level of expenditure per pupil and its allocation affected student achievement. They concluded that differences in score increases across the states also cannot be explained by resource changes alone, which may provide initial evidence that educational reform is working. Findings include:
David Grissmer, Ann Flanagan, Jennifer Kawata, and Stephanie Williamson worked at Rand Corporation at the time this research was conducted. Individually and collectively they have conducted educational research in areas such as teacher supply and demand, and the analysis of national test scores to determine the causes of changing trends. Grissmer, D. W. (1998). Education Productivity. Washington, DC: NEKIA Communications. This document is available for download from the Educational Resources Information Center (ERIC). If often appears, from what we hear and read, that the massive infusion of resources into public education has done nothing to bolster student achievement scores and that American students' scores on international assessments rank far below the scores of students from other countries. This report examines those perceptions through the lens of “education productivity.” Grissmer attempts to define productivity within the educational context and asks whether it is a useful concept to employ in evaluating educational outcomes and communicating with corporate America as well as with the general public. He cites challenges to the more traditional concept of school productivity, including the need for comprehensive tests that measure both depth and breadth of knowledge; and the issue of separating the contribution that schools make to student learning from that of families, communities, and other sources of education. He proposes instead to measure education productivity, which would include all sources of learning and supports for learning. The report concludes by noting that productivity research could yield the most important information for policymakers in education, specifically – how to use limited resources the most cost-effectively and what additional outputs would be achieved with additional resources. Grissmer also comments that such research can better inform the debate about public education, writing that “research on productivity can play a key role in restoring trust between educators and policymakers, and between the research community and the American people, who fund education.” David Grissmer is a senior management scientist at RAND. His education research includes teacher supply and demand, teacher compensation and attrition patterns, analysis of national test scores to determine the causes of changing trends, analyzing state test scores to determine causes of state differences and effects of class size reductions. |
Inside the Black Box Schooling, statistics, and poverty: Can we measure school improvement? Growth in school revisited: Achievement gains from the fourth to the eighth grade Student achievement: What state NAEP test scores tell us Education Productivity [PDF, 1.2 MB] Download the entire toolkit and more.
|
© 2006, SETDA. All rights reserved.