«® Knowledge Works Logo CMYK Equivalents - For Print Materials Acknowledgements Thanks to generous support from the Nellie Mae Education Foundation, ...»
Ensuring and Evaluating Assessment Quality | 14 State and District Roles Because districts are likely accustomed to having complete authority over their local assessment systems, and states too are accustomed to garnering sole responsibility for the state assessment and accountability system, navigating a new partnership to balance the needs of both parties must be carefully planned with newly established lines for open communication. The state needs to clearly articulate the full scope of the pilot expectations including the evidence and data collection protocols that will be necessary to support the comparability of the assessment system. While the state’s innovative pilot is still new and in the process of improving and scaling, state and district leaders must work together to find a balance between the need to collect the right evidence, and the reality of the local burden for gathering and organizing the necessary documentation. Adaptability will be a key characteristic for success in the state and district leadership as the nature and scope of the data collection will likely evolve over the course of the first few implementation years. Key comparability considerations for both state and local officials are provided in Figure 4 below.
Figure 4. Key Comparability Considerations for State and Local Officials
Ensuring and Evaluating Assessment Quality | 15
The PACE pilot gathers multiple sources of evidence to support the claims that the determinations are comparable within each district, across the different PACE districts, and with non-PACE districts. While the PACE model engages in many of the processes and audits discussed throughout this brief to establish comparability, the contrasting groups standard setting process is an additional method of achieving comparability used within PACE.
The standard setting method is used to determine the location in the score distributions of the appropriate “cut points” for establishing achievement levels. New Hampshire needed to choose a standard setting method that relied heavily on the common achievement level descriptors to ensure that the standards set across districts (both PACE and non-PACE) were comparable. An examinee-centered judgmental method called contrasting groups was used to establish cut points. This standard setting method involves judgments from panelists about the qualifications of the examinees based on prior knowledge of the examinee. PACE teachers were asked to make judgments about the achievement level that best described each of their students from the previous year.
The contrasting groups standard setting methodology then involves comparing the PACE competency scores with student placements into achievement levels in order to determine cut scores that would accurately classify the highest percentage of students.
Ensuring and Evaluating Assessment Quality | 16 The results of the standard setting method provided for comparable annual determinations by a number of measures, including a comparison with the distribution of achievement for those grade levels tested using the Smarter Balanced statewide assessment. Figure 5 below shows the percent of students deemed proficient in ELA for the PACE and Smarter Balanced grade levels for the pilot districts. While there are some small fluctuations in performance across the grade levels, they do not appear to be primarily a function of the assessment system and are not atypical from those fluctuations seen in districts administering the same assessment system across grade levels. New Hampshire is committed to ensuring comparability and continuous improvement, so trends in student achievement will be closely monitored when results from Year 2 are available.
Figure 5. PACE Districts Percent Scoring at Levels 3 & 4 in ELA
Ensuring and Evaluating Assessment Quality | 17 Summary This third brief in our series of State Readiness Conditions publications is designed to help state leaders recognize the unique comparability opportunities associated with the Innovative Assessment and Accountability Demonstration Authority flexibility afforded under the recently passed ESSA. We also present a realistic picture of the challenge associated with planning for and gathering comparability evidence within an innovation pilot. This brief clarifies the definition of comparability and provides in-depth examples of the design features and implementation processes that would support claims of comparability under a Demonstration Authority. KnowledgeWorks and the Center for Assessment will continue to support states through the summer and fall with additional briefs on topics related to fleshing out the design of a
Demonstration Authority application, including:
KnowledgeWorks can help states, districts, and other interested stakeholders establish the policy environments to support personalized learning at scale. The organization’s expertise spans the federal, state, and district levels, supporting states with strategies to leverage current policy opportunities, remove existing policy barriers, and develop new policies that will help states create an aligned policy environment to support personalized learning. To learn more, contact the following
The Center for Assessment strives to increase student learning through more meaningful educational assessment and accountability practices. We engage in deep partnerships with state and district education leaders to design, implement, and evaluate assessment and accountability policies and programs. We strive to design technically sound policy solutions to support important educational goals. The Center for Assessment’s professionals have deep expertise in educational measurement, assessment, and accountability and have applied this expertise to assessment challenges ranging from improving the quality of classroom assessments to ensuring the technical quality of state’s large-scale achievement tests and ultimately to designing coherent assessment and accountability systems.
For Assessment and Accountability System
Design and Strategic Implementation:
Scott Marion, Ph.D.
Executive Director email@example.com For Technical Quality and Comparability
Design and Analyses:
Susan Lyons, Ph.D.
Associate firstname.lastname@example.org For Assessment Quality and Performance
Jeri Thompson, Ed.D.
Senior Associate email@example.com
KnowledgeWorks is focused on ensuring that every student experiences meaningful personalized learning that allows them to thrive in college, career and civic life. We develop the capabilities of educators to implement and sustain competency-based and early Knowledge Works Logo CMYK Equivalents - For Print Materials college schools, work with state and federal leaders to establish aligned policy conditions and provide national thought leadership around the future of learning. Our subsidiary, StriveTogether, manages the national cradle-to-career network and helps communities to improve student outcomes through collective impact. www.knowledgeworks.org The National Center for the Improvement of Educational Assessment, Inc. (Center for Assessment) is a Dover, NH based not-for-profit (501(c)(3)) corporation that seeks to improve the educational achievement of students by promoting enhanced practices in educational assessment and accountability. The Center for Assessment does this by providing services directly to states, school districts, and other organizations regarding the design, implementation, and evaluation of assessment and accountability systems.
As a non-profit organization committed to the improvement of student learning, the Center for Assessment maintains a strong “open-source” ethic in terms of distributing its many creations and inventions. For example, the Center has developed many tools related to alignment methodology, student growth analyses, student learning objectives, comparability methods for innovative assessment systems, and validity evaluation that it provides freely to its clients and other non-commercial entities. www.nciea.org The Nellie Mae Education Foundation is the largest philanthropic organization in New England that focuses exclusively on education. The Foundation supports the promotion and integration of student-centered approaches to learning at the middle and high school levels across New England—where learning is personalized; learning is competency-based; learning takes place anytime, anywhere; and students exert ownership over their own learning. To elevate
student-centered approaches, the Foundation utilizes a four-part strategy that focuses on:
building educator ownership, understanding and capacity; advancing quality and rigor of SCL practices; developing effective systems designs; and building public understanding and demand. Since 1998, the Foundation has distributed over $180 million in grants. For more information about the Nellie Mae Education Foundation, visit www.nmefoundation.org.