SETDA's NLI Toolkit 2003 - Data Driven Decision-Making & Data Collection


	1. Chapter Summary 2. Description of the National Context 3. Implementation Strategies 4. Using Data for Student Achievement 5. Promising Practices 6. Resources & Readings 7. Next Steps

Next | Previous | Chapter Home | Toolkit Home

Description of the National Context for Data Driven Decision-Making

The Focus on Data Driven Decision-Making

One could make the argument that our journey into data driven decision-making begins with Galileo and the Renaissance period of the 16th and 17th centuries. It was then that scientists and philosophers of the day began to increasingly rely upon observations of their surroundings to better understand their world (Shadish, Cook, & Campbell, 2002). This reliance upon empirical data, or observed data, revolutionized the world these scientists knew. It revealed trends and phenomena that unlocked knowledge long hidden in the seemingly indecipherable noise of life's experiences. To a large degree, we, as educators and administrators, are continuing their work when we pour over the latest achievement data in order to determine strategies for improving our students' education.

While the word accountability has replaced discovery, we are still mining data for the information that will help us better understand our world. This information will empower us to help our students "develop the intellectual tools and learning strategies needed to acquire the knowledge that allows people to think productively about history, science and technology, social phenomena, mathematics, and the arts" (National Research Council, 2000, p. 5). While this may not seem as romantic or groundbreaking as the unveiling of Galileo's heliocentric universe, the members of SETDA believe it is no less important.

The goal of this toolkit is to provide state educational technology directors with a series of tools that will help them collect, analyze, and use data, as well as report their findings. In order to do this, we had to ask ourselves some important questions: What does data driven decision-making mean? Does it mean the same thing to different states? What are the data collection and analysis capabilities of each of the states? How can we create a document that will serve all states? What role do we as state educational technology directors play in the overall trend towards data analysis throughout education? What role does technology play in data driven decision-making? What is the effect of technology integration on student learning? Are the funds being used for the integration of projects that we know work? If we do not know what technology interventions work, are we taking steps to determine their efficacy? What data elements do we need to collect in order to inform our goals?

States have a variety of data collection systems ranging from comprehensive statewide systems to very minimal collection of data at a local level. Large and meaningful differences exist among the different states' capabilities, needs, and resources in collecting and using this data. In addition, once collected, this data may have varying uses and implications for the different levels of stakeholders. Despite these differences, all the states are united in their need for accurate and reliable data collection and analysis in order to improve their students' achievement.

While many educators may think that this is a new conversation instigated by the passage of the No Child Left Behind Act (NCLB), in fact, it reflects a larger societal trend. In nearly every national and international industrial sector, the world is witnessing the ability of large networks to collect data and, therefore, transform the understanding of how enterprises successfully function.

While technology was a drain on productivity in its early adoption in nearly every industrial sector, we are now seeing a dramatic ability to use data for informed decision-making. Everything from how Wal-Mart tracks inventory to the way Baldridge Award recipient school districts implement reform is benefited by the use of timely and relevant data for continuous improvement. Despite this international trend, many within the education community believe that the goals of the NCLB legislation are unattainable without the adoption of more powerful uses of data to track adequate yearly progress.

The NCLB, Title II, Part D legislation calls for increased academic achievement through strategic, effective approaches to the use of technology by schools. Given this directive, it is clear that the data collection processes used by most states in the past, school and district surveys, is not sufficient. The process must include data from teachers and students at the classroom level-in addition to state, district, and school survey data that address policies, practices, and impact.

Stakeholders in Data Collection Efforts

The identification and collaboration of state, district, and school level stakeholders are critical to constructing a data collection process that will meet the NCLB directive outlined above. Several different stakeholders are in need of critical school data. Six major categories identified by SETDA are:

Federal-U.S. Department of Education
Individual States-State Departments of Education
School Districts
Schools
Community Members
Other Stakeholders (i.e., legislators, parents, and students)

In each of the six levels, there are different data and sources of collection to address various needs.

SETDA's Common Data Elements Initiative

In 2002, SETDA set out to identify a set of common data elements (CDE) for assessing progress in educational technology throughout the nation. The purposes for the data include: to track state progress on NCLB, Title II, Part D and to assist states and districts in decision-making. The CDE initiative represents consensus and collaboration among states. The challenge to agree upon a set of common data elements is difficult; however, states have been able to reach consensus under the guidance of the CDE Task Force and through the active participation of additional state educational technology directors throughout the entire process. SETDA chose this initiative as one of the topics of focus during the NLI 2002, which helped ensure greater participation and collaboration among state directors. Although flexibility on the part of the different states has been an important facet of the CDE initiative throughout the process, the states have been able to agree upon what key questions, indicators, and data elements truly respond to NCLB and the needs of the states, which were included in SETDA's NLI 2002 Toolkit.

Throughout 2003, SETDA's Data Collection Committee and the CDE Task Force developed and implemented a State Profile Survey, completed by 46 states and the District of Columbia, to provide information on the states' implementation of the NCLB Title II, Part D programs. The National State Profile Report is available at www.setda.org.

Additionally, the CDE Task Force, in conjunction with the Metiri Group, began developing state- and district-level surveys. SETDA is currently in the process of piloting the district- and building-level surveys, as well as a suite of observation tools developed during 2003. The NLI 2003 Data Work Group reviewed and provided additional input to the CDE Task Force and the Metiri Group for the continued development of these tools. Collaboration and coordination among the states have been driving factors throughout the CDE initiative.

SETDA is trying to narrow the scope of what data is collected, so as to much more specifically relate it to the integration of educational technology. The scope would encompass the needs of the various stakeholders to address successful integration of educational technology. Given the fact that this focus on educational technology cannot answer all of the questions or needs of the comprehensive task of improving student achievement, the data collected will address issues surrounding student achievement with the use of technology.

Factors for Consideration

There are several factors to consider about the collection of data relating to educational technology:

The U.S. Department of Education needs information about the uses and results of educational technology funding to validate the worth of the programs and to report to Congress for continued allocation.
The U.S. Department of Education is currently implementing the Performance-Based Data Management Initiative (PBDMI). The PBDMI is an attempt to increase focus on outcomes and accountability, to reduce data collection redundancy within the overall system, and to create partnerships based on common data collection goals. The U.S. Department of Education is attempting to make this initiative comprehensive across all educational programs. In addition, they are determining what will be collected in the area of instructional technology, which is only one of the content areas.
The U.S. Department of Education has commissioned the National Educational Technology Trends Study (NETTS), which will include data about educational technology funding. NETTS will evaluate the NCLB, Title II, Part D grant program. The results from this study will be used by administrators and policy-makers from the federal, state, and local level to inform decision-making.
The Government Performance and Results Act (GPRA) is another set of indicators on which the U.S. Department of Education must respond, including indicators relating to educational technology. These indicators are important for consideration as decisions are made regarding the data that will be collected.
States, districts, schools and other entities also need data to inform decisions about the integration of technology and student achievement.

Despite the transformative power of data use seen in many sectors throughout the world, SETDA realizes that data driven decision-making is not a panacea for the many challenges faced by our nation's educational system. However, while resisting "data-driven mania," we must capitalize on the potent and vital nexus that exists between technology and data to create long-term reform within our schools (Love, 2002, p. xxiii).