WWW.THESES.XLIBX.INFO
FREE ELECTRONIC LIBRARY - Theses, dissertations, documentation
 
<< HOME
CONTACTS



Pages:   || 2 | 3 |

«® Knowledge Works Logo CMYK Equivalents - For Print Materials Acknowledgements Thanks to generous support from the Nellie Mae Education Foundation, ...»

-- [ Page 1 ] --

ADDRESSING

ACCOUNTABILITY

ISSUES

INCLUDING

COMPARABILITY

in the Design and Implementation

of an Innovative Assessment and

Accountability System

July 2016

®

Knowledge Works Logo

CMYK Equivalents - For Print Materials

Acknowledgements

Thanks to generous support from the Nellie Mae Education Foundation, KnowledgeWorks, and the National Center for the Improvement of Educational Assessment (Center for Assessment) have partnered to help states better understand and leverage the new Innovative Assessment and Accountability Demonstration Authority authorized under the Every Student Succeeds Act (ESSA). The goal of this partnership is to help states identify and explore a set of readiness conditions that are critical to the development of a high quality application and implementation process under this new authority. While we share a history of advocacy for next generation assessments, our organizations each bring a unique perspective to this work. KnowledgeWorks focuses on policy development, partnering with states, districts, and educators to identify and remove policy barriers that inhibit the growth of personalized learning. The Center for Assessment specializes in the design of assessment and accountability systems, helping states, districts, and other entities improve the quality of these systems and maximize student success.

Lyons, S., Marion, S.F., Pace, L., & Williams, M. (2016). Addressing Accountability Issues including Comparability in the Design and Implementation of an Innovative Assessment and Accountability System. www.knowledgeworks.org and www.nciea.org.

Ensuring and Evaluating Assessment Quality | 2 Table of Contents Introduction 4 Purpose 5 Alignment to Theory of Action 6 Defining Comparability 7 Comparability by Design 9 Methods for Establishing a Strong Evidence Base to Support Claims of Comparability State and District Roles 15 State Example 16 Summary 18 Additional Support 19 About 20 KnowledgeWorks 20 National Center for the Improvement of Educational Assessment 20 Nellie Mae Education Foundation 20 Ensuring and Evaluating Assessment Quality | 3 Introduction This is the third in a series of policy and practice briefs produced by KnowledgeWorks and the National Center for the Improvement of Educational Assessment (Center for Assessment) designed to assist states in thinking through the opportunities and challenges associated with flexibility provided under the Every Student Succeeds Act (ESSA).1 These briefs help define “Readiness Conditions” for states considering applying for and successfully implementing an innovative assessment and accountability system as defined by the Demonstration Authority opportunity under ESSA. In addition to those that have already been published, the following briefs will be released over

the next few months:

Supporting Educators and Students through Implementation of an Innovative Assessment and Accountability System Evaluating and Continuously Improving an Innovative Assessment and Accountability System Establishing a Timeline and Budget for Design and Implementation of an Innovative Assessment and Accountability System Building Capacity and Stakeholder Support for Scaling an Innovative Assessment and Accountability System Brief #3 in a series of policy and practice briefs designed to help states prepare for the ESSA Assessment and Accountability Demonstration Authority. We are grateful to the Nellie Mae Foundation for their generous support of this project.

Ensuring and Evaluating Assessment Quality | 4 Purpose The Innovative Assessment and Accountability Demonstration Authority (hereafter known as the “innovative pilot” or the “Demonstration Authority”) provides states with an opportunity to collaborate with a sample of local districts to pilot a new kind of assessment and accountability system within the state. This system does not have to rely on statewide, standardized assessments as the sole indicator of student achievement, but instead may pilot different types of non-standardized assessments (e.g., instructionally embedded assessments, performance tasks) that may provide for some degree of local flexibility. Because states must incorporate assessment results from the pilot districts into the state accountability system alongside the results generated from the non-pilot districts, the assessment system must meet all of the same technical requirements as the state standardized assessments—e.g., alignment, validity, reliability, accessibility.2 Additionally, because the innovative pilot will take time to scale statewide, the state must ensure that the assessment systems are producing comparable results within pilot districts, among pilot districts, and importantly, across pilot and non-pilot districts.

The purpose of this brief is to support states in planning for a successful Demonstration Authority application by providing key conceptual and technical considerations related to promoting and evaluating comparability in an innovative assessment and accountability pilot. We begin with a discussion of alignment to the state’s theory of action so the pilot focuses on the intended goals of the system.





Next, we define comparability in the era of ESSA flexibility, and lay the groundwork for a common understanding of how evidence of comparability differs depending on the nature and use of the reported scores. We then delve deeply into how states could approach comparability from a design perspective, providing detailed examples of processes that states could use to support their intended comparability claims. We additionally provide descriptions of the state and local roles for ensuring comparability. Lastly, we provide a case study that details a key comparability practice from the innovative assessment and accountability system in New Hampshire.

For detailed information regarding the technical quality considerations of an innovative pilot, please refer to Brief #2 Ensuring and Evaluating Assessment Quality in the Design and Implementation of an Innovative Assessment and Accountability System.

Ensuring and Evaluating Assessment Quality | 5 Alignment to Theory of Action As emphasized in the Project Narrative brief, the importance of a clear articulation of the state vision and the associated theory of action for attaining that vision cannot be overemphasized. Comparability is a critical goal whenever assessments are being used for accountability. This is especially true when states have incorporated some degree of local flexibility into its assessment systems. Providing for comparability within the initial design conceptualization of the system will be crucial to the success of the pilot.

In order to design and administer meaningful assessment that will change the way instruction and learning occurs in the classroom, local educators will need to engage in rich discussions about what deep learning looks like for every grade level and content area. For example, defining the expectations for student performance in a competency-based education model requires that educators across the state have shared definitions about both the content standards and the required evidence for evaluating student competence relative to the content standards. Therefore, the beginnings of a comparability argument are baked into the learning system of the innovative pilot that the assessment and accountability systems must capture.

This brief provides examples of how states can achieve the goal of comparability by planning for it in the pilot design and the processes and audits that comprise the new assessment and accountability system. Each of these design features should be born out of an alignment with the overall theory of action for how learning is changing within the state, and how the pilot will ultimately bring about that change.

Ensuring and Evaluating Assessment Quality | 6 Defining Comparability In educational measurement, comparability is usually premised on the notion of score interchangeability. If scores can be used interchangeably, that means the scores support the same interpretations about what students know and can do relative to the assessed content. Comparability is an accumulation of evidence to support claims about the meaning of test scores and whether scores from two or more tests can be used to support the same inferences and uses. While it is typical in the United States to support comparability by standardizing testing conditions (e.g., administration, scoring), we must acknowledge that score comparability is not necessarily at odds with flexibility.3 As an example, we provide accommodations for standardized assessments because we believe this type of “flexibility” actually improves our claims of score comparability by removing barriers to the assessed content. Just like changing the administration conditions for students with different abilities supports our notion of comparability, it could be argued that changing the mode of assessment (e.g., performance-based assessment) will provide better information about what students know and can do for students in different educational settings (e.g., competencybased), than we could glean from traditional standardized assessments.

–  –  –

Because claims of comparability are inherently tied to the interpretations and uses of the scores, comparability rests on what is being reported. This means that evidence used to support claims of comparability will differ depending on the nature (or grain-size) of the reported scores. For example, supporting claims of raw score interchangeability—the strongest form of comparability—would likely require the administration of a single assessment form with measurement properties that are the same across all respondents (i.e., measurement invariance). Any state assessment system with multiple assessment forms fails to meet this level of score interchangeability. Instead, the design of most state assessment systems aims to be comparable enough to support scale score interchangeability. This level of comparability typically requires that multiple test forms are designed to the same blueprint, administered under almost identical conditions, and scored using the same rules and procedures. Still, many states continue to struggle to meet this level of comparability (e.g., challenges with multiple modes of administration—paper-based, computer-based, and device-based). In this way, comparability is an evidence-based argument, and the strength of evidence needed will necessarily depend on the type of score being supported. As shown in Figure 1, comparability lies on a continuum that is based on both the degree of similarity in the assessed content and the granularity of the score being reported.4 Gong, B., & DePascale, C. (2013). Different but the same: Assessment “comparability” in the era of the Common Core State Standards. Washington, DC: The Council of Chief State School Officers.

Winter, P. (2010). Comparability and test variations. In P. Winter (Ed.), Evaluating the comparability of scores from achievement test variations (pp. 1–11). Washington, DC: Council of Chief State School Officers.

–  –  –

The Demonstration Authority requires states to ensure that summative “annual determinations” (e.g., performance levels) are comparable. Comparability, therefore, must exist at the level of the annual determinations. This means that if a student is determined to be “proficient” relative to the grade-level content standards in one district in the state, had that student been assigned to another district’s assessment system (either pilot or non-pilot) he or she could expect to be proficient. To support claims of comparability at the annual determination level, any pilot program will need to build in a number of processes and auditing mechanisms to create a strong evidence base for supporting the claims of comparability within each pilot district, among pilot districts, and across pilot and non-pilot districts.

Winter, P. (2010). Comparability and test variations. In P. Winter (Ed.), Evaluating the comparability of scores from achievement test variations (p. 5). Washington, DC: Council of Chief State School Officers.

Ensuring and Evaluating Assessment Quality | 8 Comparability by Design The methods for gathering evidence to support a comparability claim are not a series of analyses, but rather begin with the design of the innovative assessment and accountability pilot itself. In traditional standardized assessment programs, comparability is generally established by planning for it in the assessment system design (e.g., addressing the same learning targets in the same ways, embedding items), evaluating the degree of comparability achieved (e.g., analyses of differential item functioning), and then, if necessary, adjusting the measurement scales to account for differences (e.g., equating). Providing evidence of comparability for the innovative assessment system will require discussion related to each of these steps, even if the methods related to each step are necessarily different. Three key questions shown in Figure 2 below can guide the process of designing a pilot to produce comparability results—comparability by design.

Figure 2. Comparability by Design—Guiding Questions

–  –  –

The order of these guiding questions is exceedingly important. It will not be possible to evaluate the degree of comparability that these scores produce under different assessment systems if comparability has not been carefully planned for (e.g., through common items or tasks). Similarly, it will not be possible to calibrate results if the nature and magnitude of the adjustments are not known through a careful evaluation of the degree of comparability achieved across assessment systems. No amount of evaluation and calibration can fix a system that has not been carefully designed to produce scores that are likely to be comparable. Thus, garnering evidence to support comparability of the assessment system results will require thoughtful planning of the program processes that will promote comparability, and the program monitoring mechanisms that will evaluate comparability. Examples of how this could be done to support claims of comparability of results within pilot districts, among pilot districts, and across pilot and non-pilot districts are provided on the next few pages.



Pages:   || 2 | 3 |


Similar works:

«Medicare Claims Benefit Manual Chapter 15 – Covered Medical and Other Health Services “Incident To” Revision Date 11/18/11 60 Services and Supplies 60.1 Incident To Physician’s Professional Services 60.2 Services of Nonphysician Personnel Furnished Incident To Physician’s Services 60.3 Incident To Physician’s Services in Clinic 60.4 Services Incident to a Physician’s Service to Homebound Patients Under General Physician Supervision Medicare Claims Processing Manual Chapter 12...»

«Library of Congress  Albert J. Beveridge  MARCH OF THE FLAG Address to an Indiana Republican Meeting Indianapolis, Indiana, 16 September 1898 Excerpts * F ellow citizens,  it is a noble land that God has Albert Beveridge, 1900 given us; a land that can feed and clothe the world; a land whose coastlines would enclose half the countries of Europe; a land set like a sentinel between the two imperial oceans of the globe, a greater England with a nobler destiny. It is a mighty people that...»

«Sacred and Profane: The Common Roots of Theatre and Worship Pamela R. Hendrick Sunday, August 16, 2009 You’ve heard my professional theatre credentials, so just to give you a spiritual point of reference, I am speaking today as someone who has always considered herself to be a devout agnostic. But more recently, I’ve come to think of myself as a spiritual humanist. And I realize that that perspective has usually been present in my theatrical work. A while ago I was asked if I would talk...»

«In the Name of the Son (and the Daughter): Intergenerational Mobility in the United States, 1850-1940.∗ Claudia Olivetti and M. Daniele Paserman Boston University and NBER April 2015 Abstract This paper estimates historical intergenerational elasticities between fathers and children of both sexes in the United States using a novel empirical strategy. The key insight of our approach is that the information about socio-economic status conveyed by first names can be used to create pseudo-links...»

«Banishing the Spirits 1 © 1997 Max Dashu Advance excerpt from Secret History of the Witches They that burn you for a witch lose all their coals. [Denham Tracts, 84] Folk memory never relinquished its affection for the ancient religions. Sometimes it gave voice to a wistful hope for the deliverance of the old ways. The beloved old deities, it was said, were cursed to wander as spectres. Their banning could only be lifted under very particular, difficult circumstances, which tradition alludes to...»

«Containment: Failures and Successes during the Kennedy Administration Unit Length: 5 days (55 minutes periods) 11th Grade US History By: Dawn Bushyeager Overview/Goals This unit is designed to help students with a large Cold War culminating Classroom Based Assessment about the US foreign policy of containment from 1945-1975.This unit will help students answer the following essential questions: • Why was the policy of containment used during the Kennedy administration? • Using specific...»

««Studi Emigrazione/Migration Studies», XLV, n. 169, 2008. La figura dell’immigrato nel cinema statunitense. Quando la passione cinefila raggiunge l’analisi sociale Lo schermo e la riscoperta dell’ “etnicità” A partire dalla fine degli anni 1970, la riscoperta dell’“etnicità” nella società e nella storiografia statunitensi spinge alcuni studiosi ad interessarsi del modo in cui sono stati e sono rappresentati i gruppi immigrati più consolidati1. Questa ricerca non ha...»

«Revista de Claseshistoria Revista Publicación digital de Historia y Ciencias Sociales Índice de Autores Artículo Nº 352 Claseshistoria.com 15 de marzo de 2013 ISSN 1989-4988 DEPÓSITO LEGAL MA 1356-2011 RUBÉN DOMÍNGUEZ MÉNDEZ La historia de la URSS en la visión de Martin Malia: comprender la Revolución Rusa RESUMEN El artículo nos acerca a la obra Comprender la Revolución rusa perteneciente al historiador estadounidense Martin Malia. Tras pasar largas temporadas en la URSS...»

«Cencerradas, cultura moral campesina y disciplinamiento social en la España del Antiguo Régimen Tomás A. Mantecón Movellán Mundo Agrario, vol. 14, nº 27, diciembre 2013. ISSN 1515-5994 http://www.mundoagrario.unlp.edu.ar/ DOSSIER Cencerradas, cultura moral campesina y disciplinamiento social en la España del Antiguo Régimen (1) Rough music, moral peasant culture and social discipline in Old Regime Spain Tomás A. Mantecón Movellán (*) Universidad de Cantabria (España) Dpto. de...»

«Marcas Blancas Nuevo concepto comercial del sector alimentario Marcas Blancas Nuevo concepto comercial del sector alimentario Sandra Florensa Alcaraz Clàudia Fradera Fontcuberta Mónica Frau Pons Treball d’Economia aplicada II 2n ADE/ECO, curs 2008-09 Professor: Joan Ribas-Tur Facultat de Ciències Econòmiques i Empresarials Universitat Pompeu Fabra Marcas Blancas Nuevo concepto comercial del sector alimentario Contenido Prólogo 5 Agradecimientos 8 1ª Parte: Análisis teórico 9...»

«C on du c tin g S tu di e s C on fe re n ce 20 1 6 24 -26 Ju ne, S t An ne’ s College CONFERENCE SCHEDULE PREVIEW (31 May 2016) Friday 24 th June LOCATION Mary Ogilvie Lecture Theatre (MOLT) Marquee 11:30 Registration opens (MOLT Foyer) 12:45 Welcome 13:00 Invited Introductory Talk: Sian Edwards (Head of Conducting, Royal Academy of Music) 13:30 Session 1a: Historical Perspectives Session 1b: Leadership 1 Prof Raymond Holden, Chair Cayenna Ponchione-Bailey, chair Simone Laghi (Cardiff...»

«INDICE Dedicatorias Prólogo PRIMERA PARTE EL DR. RYKE GEERD HAMER: HISTORIA DE UN PIONERO LAS 5 LEYES FUNDAMENTALES o LA MEDICINA PATAS ARRIBA Primera ley: La ley de hierro del cáncer «El trauma es el detonante» Segunda ley: Las dos fases de la enfermedad «Nada existe sin su contrario» Tercera ley: El sistema ontogenético de los tumores y de las enfermedades equivalentes «Más allá de la complejidad, todo es simple» Cuarta ley: El sistema ontogenético de los microbios «Trabajadores...»





 
<<  HOME   |    CONTACTS
2016 www.theses.xlibx.info - Theses, dissertations, documentation

Materials of this site are available for review, all rights belong to their respective owners.
If you do not agree with the fact that your material is placed on this site, please, email us, we will within 1-2 business days delete him.