FR Doc E9-27426[Federal Register: November 18, 2009 (Volume 74, Number 221)] [Rules and Regulations] [Page 59687-59834] From the Federal Register Online via GPO Access [wais.access.gpo.gov] [DOCID:fr18no09-9] Download:----------------------------------------------------------------------- [[Page 59687]] ----------------------------------------------------------------------- Part III Department of Education ----------------------------------------------------------------------- 34 CFR Subtitle B, Chapter II Race to the Top Fund; Final Rule [[Page 59688]] ----------------------------------------------------------------------- DEPARTMENT OF EDUCATION 34 CFR Subtitle B, Chapter II [Docket ID ED-2009-OESE-0006] RIN 1810-AB07 Race to the Top Fund AGENCY: Department of Education. ACTION: Final priorities, requirements, definitions, and selection criteria. ----------------------------------------------------------------------- Catalog of Federal Domestic Assistance (CFDA) Number: 84.395A. SUMMARY: The Secretary of Education (Secretary) announces priorities, requirements, definitions, and selection criteria for the Race to the Top Fund. The Secretary may use these priorities, requirements, definitions, and selection criteria in any year in which this program is in effect. DATES: Effective Date: These priorities, requirements, definitions, and selection criteria are effective January 19, 2010. FOR FURTHER INFORMATION CONTACT: James Butler, U.S. Department of Education, 400 Maryland Ave., SW., room 3E108, Washington, DC 20202- 6400. Telephone: 202-205-3775 or by e-mail: racetothetop@ed.gov. SUPPLEMENTARY INFORMATION: Purpose of Program: The purpose of the Race to the Top Fund, a competitive grant program, is to encourage and reward States that are creating the conditions for education innovation and reform; achieving significant improvement in student outcomes, including making substantial gains in student achievement, closing achievement gaps, improving high school graduation rates, and ensuring student preparation for success in college and careers; and implementing ambitious plans in four core education reform areas-- (a) Adopting internationally benchmarked standards and assessments that prepare students for success in college and the workplace; (b) Building data systems that measure student success and inform teachers and principals about how they can improve their practices; (c) Increasing teacher and principal effectiveness and achieving equity in their distribution; and (d) Turning around our lowest-achieving schools. Additional information on the Race to the Top program can be found at: http://www.ed.gov/programs/racetothetop. Program Authority: American Recovery and Reinvestment Act of 2009, Division A, Section 14006, Public Law 111-5. We published a notice of proposed priorities, requirements, definitions, and selection criteria (NPP) for this program in the Federal Register on July 29, 2009 (74 FR 37804). That notice contained background information and our reasons for proposing the particular priorities, requirements, definitions, and selection criteria. There are a number of differences between the NPP and this notice of final priorities, requirements, definitions, and selection criteria as discussed in the Analysis of Comments and Changes section elsewhere in this notice. Public Comment: In response to our invitation in the NPP, 1,161 parties submitted comments on the proposed priorities, requirements, definitions, and selection criteria. Generally, we do not address technical and other minor changes, nor do we address suggested changes that the law does not authorize us to make under the applicable statutory authority. In addition, we do not address general comments that raised concerns not directly related to the NPP. Introduction The Race to the Top program, a $4.35 billion fund created under the American Recovery and Reinvestment Act of 2009 (ARRA), is the largest competitive education grant program in U.S. history. The Race to the Top Fund (referred to in the ARRA as the State Incentive Grant Fund) is designed to provide incentives to States to implement large-scale, system-changing reforms that result in improved student achievement, narrowed achievement gaps, and increased graduation and college enrollment rates. The ARRA specifies that applications for Race to the Top funds must address the four assurance areas referenced in section 14006(a)(2): Enhancing standards and assessments, improving the collection and use of data, increasing teacher effectiveness and achieving equity in teacher distribution, and turning around struggling schools. The Department published the NPP to solicit public comment on the priorities, requirements, and selection criteria that State applications will address in accordance with this statutory requirement. The NPP prompted an outpouring of public comments. Some 1,161 commenters submitted thousands of unique comments, ranging from one paragraph to 67 pages. Parents submitted comments, as did professional associations. From the statehouse to the schoolhouse, scores of public officials and educators, governors, chief State school officers, teachers, and principals weighed in with suggestions and critiques. All told, individuals from all 50 States and the District of Columbia, including over 550 individuals and 200 organizations, commented on the NPP. The extensive and thoughtful public commentary on the NPP has been invaluable in helping the Department revise, improve, and clarify the priorities, requirements, definitions, and selection criteria for the Race to the Top program. A discussion of the most significant changes follows. Major Changes in the Selection Criteria, Priorities, Requirements, and Definitions State Success Factors Many of the commenters expressed concern that the NPP's encouragement of comprehensive and coherent statewide reform was undercut by the need for State applicants to organize their plans around each of the four reform assurances, one at a time. In response to this concern, the Department has reorganized a number of the criteria, moving key criteria from the Overall section to a new section at the beginning of the selection criteria called State Success Factors. This new section provides States with the opportunity to start their proposals with clear statements of their coherent, coordinated, statewide reform agendas. As several commenters noted, States face at least three overarching issues critical to their success in implementing their Race to the Top plans--the need for a coherent reform agenda, the capacity to lead LEAs, and the ability to improve outcomes. In this notice, these three issues are reflected in the State Success Factors as follows: Criterion (A)(1) pertains to a State's ability to articulate a comprehensive and coherent education reform agenda, and to engage its local educational agencies (LEAs) in strongly committing to and participating in that agenda; criterion (A)(2) relates to a State's capacity to implement its proposed plans through strong leadership, successfully supporting its LEAs in improving student outcomes, administering a grant of this magnitude efficiently, and organizing its financial resources to optimize impact; and finally, criterion (A)(3) asks States to demonstrate their ability to significantly improve education outcomes for students across the State. More specifically, criterion (A)(1)(i) is a new criterion that asks States to set forth a comprehensive and coherent reform agenda that clearly articulates their goals for implementing reform in [[Page 59689]] the four education areas described in the ARRA and improving student outcomes statewide, establishes a clear and credible path to achieving these goals, and is consistent with the specific reform plans that the State has proposed throughout its application. Under criterion (A)(1)(ii) (proposed criterion (E)(3)(iv)), States will demonstrate the participation and commitment of their LEAs. First, as described in criterion (A)(1)(ii)(a), the strength of LEAs' commitments to their State's plans will be evaluated based on the terms and conditions in a State's binding agreements with its LEAs. (To support States' efforts, the Department has drafted a model Memorandum of Understanding (MOU) and included it in Appendix D of this notice.) Criterion (A)(1)(ii)(b) has been added to make it clear that the commitment of participating LEAs will also be judged, in part, based on LEAs' agreements to implement all or significant portions of the work outlined in the State's plan. Criterion (A)(1)(ii)(c) clarifies that the extent of an LEA's leadership support for participating in the State's Race to the Top plans will be assessed by how many signatures are on the binding agreement between the State and the LEA, from among (if applicable) the superintendent, the president of the local school board, and the local teachers' union leader, or their equivalents (provided that there is at least one authorized LEA signatory on the agreement). For all of these criteria, States will be asked to provide as evidence examples of their participating LEA agreements as well as tables that summarize which portions of the State plans LEAs are committing to implement and how extensive the LEAs' leadership support is. Criterion (A)(1)(iii) (adapted from proposed criteria (E)(3)(iv) and (E)(4)) asks States to describe how the engagement of those LEAs that are participating in the State's Race to the Top plans will translate into broad statewide impact on student outcomes, including increasing achievement and decreasing achievement gaps for (at a minimum) reading/language arts and mathematics on the National Assessment of Educational Progress (NAEP) and on the assessments required under the Elementary and Secondary Education Act of 1965, as amended (ESEA); and increasing high school graduation rates, college enrollment rates, and college credit accumulation. Criterion (A)(2) asks States to describe their capacity to implement, scale up, and sustain their proposed plans. Criterion (A)(2)(i) (adapted from proposed criterion (E)(5)) concerns States' capacity to implement their plans. Criterion (A)(2)(i)(a) asks States to demonstrate that they have strong leadership and dedicated teams to implement their statewide education reform plans; and criterion (A)(2)(i)(b) (proposed (E)(5)(ii)) encourages States to describe the activities they will undertake in supporting participating LEAs in successfully implementing their plans. Criterion (A)(2)(i)(c) (proposed criterion (E)(5)(i)) asks States about the effectiveness and efficiency of their operations and processes for implementing a Race to the Top grant. Criterion (A)(2)(i)(d) (proposed (E)(5)(v)) further clarifies that States will be evaluated based on how they plan to use the funds for this grant, as described in their budgets and accompanying budget narratives, to accomplish their plans and meet their performance targets. Proposed criterion (E)(5)(iv), regarding collaboration between States, is not included in this final notice. In criterion (A)(2)(ii) (proposed (E)(3)(i) and (E)(3)(ii)), States demonstrate that they have a plan to use the support from a broad group of stakeholders to better implement their reform plans. Criterion (A)(2)(ii)(a) concerns enlisting the support of teachers and principals as key stakeholders. Criterion (A)(2)(ii)(b) asks States to describe the strength of statements and actions of support from other critical stakeholders, and examples of these are listed. Proposed criterion (E)(3)(iii), regarding the support of grant-making foundations and other funding sources, is not included in this final notice. Criterion (A)(3) addresses the extent to which the State has demonstrated significant progress in raising achievement and closing gaps. Criterion (A)(3)(i) (proposed criteria (E)(1)(i) and (E)(1)(ii)) provides for the evaluation of States based on whether they have made progress in each of the four education reform areas over the past several years and used ARRA and other Federal and State funding to pursue such reforms. Criterion (A)(3)(ii) (proposed criterion (E)(1)(iv)) addresses States' track records of increasing student achievement, decreasing achievement gaps, and increasing graduation rates. When evaluating these student academic outcomes, reviewers will examine student assessment results in reading/language arts and mathematics, both on the NAEP and on the assessments required under the ESEA; progress will be considered for each subgroup as well as for the ``all students'' group. Standards and Assessments In response to comments indicating that some States would have difficulty meeting a June 2010 deadline for adopting a new set of common, kindergarten-to-grade-12 (K-12) standards, this notice extends the deadline for adopting standards as far as possible, while still allowing the Department to comply with the statutory requirement to obligate all Race to the Top funds by September 30, 2010. As set forth in criterion (B)(1)(ii), the new deadline for adopting a set of common K-12 standards is August 2, 2010. States that cannot adopt a common set of K-12 standards by this date will be evaluated based on the extent to which they demonstrate commitment and progress toward adoption of such standards by a later date in 2010 (see criterion (B)(1) and Appendix B). Evidence supporting the State's adoption claims will include a description of the legal process in the State for adopting standards, and the State's plan, current progress against that plan, and timeframe for adoption. For criteria (B)(1) and (B)(2) (proposed criteria (A)(1) and (A)(2), respectively), regarding the development and adoption of common, high-quality standards and assessments, the term ``significant number of States'' has been further explained in the scoring rubric that will be used by reviewers to judge the Race to the Top applications (see Appendix B). The rubric clarifies that, on this aspect of the criterion, a State will earn ``high'' points if its consortium includes a majority of the States in the country; it will earn ``medium'' or ``low'' points if its consortium includes one-half or fewer of the States in the country. Further, for criterion (B)(2), concerning the development and implementation of common, high-quality assessments, States will be asked to present, as evidence, copies of their Memoranda of Agreement showing that the State is part of a consortium that intends to develop high-quality assessments aligned with the consortium's common set of standards. This is similar to the evidence required for criterion (B)(1) concerning the development and adoption of common standards. Finally, this notice clarifies the language in criterion (B)(3) (proposed criterion (A)(3)) regarding the transition to enhanced standards and high-quality assessments; the criterion now lists a number of activities in which States or LEAs might engage as they work to translate the standards and assessments into classroom practice. [[Page 59690]] Data Systems to Support Instruction The data systems selection criteria in the Race to the Top competition involve two types of data systems--statewide longitudinal data systems and instructional improvement systems. While numerous comments addressed the Department's emphasis on statewide longitudinal data systems in the NPP, the Department intends to give equal priority in this program to using instructional data as a critical tool for teachers, principals, and administrators to identify student needs, fill curriculum gaps, and target professional development. The final selection criteria, therefore, place significant emphasis on using data to inform professional development and fostering a culture of continuous improvement in schools and LEAs. More specifically, the final notice contains new language in criterion (C)(3)(i) (proposed (B)(3)(i)) that clarifies that this criterion concerns local instructional improvement systems, not statewide longitudinal data systems, and further clarifies the LEA's role in the acquisition, adoption, and use of local instructional improvement systems. New criterion (C)(3)(ii) was added to encourage LEAs and States to provide effective professional development on using data from these systems to support continuous instructional improvement. Great Teachers and Leaders The teachers and leaders criteria are built on two core principles that remain consistent with the NPP--that teacher and principal quality matters, and that effective teachers and principals are those whose students grow academically. Thus, this notice continues to include criteria directed at improving teacher and principal effectiveness and at ensuring that highly effective teachers and principals are serving in the high-poverty, high-minority schools where their talents are needed the most. In addition, this notice continues to define effective teachers and principals as those whose students make significant academic growth. While the final notice reaffirms these core principles, it also includes a number of changes to the criteria and related definitions based on public input. The Department received over 400 comments in this reform area, many of which provided helpful suggestions that informed our revisions. One commenter suggested that the greatest contribution that the Race to the Top program could make would be to encourage the development of outstanding models for teacher and principal evaluation systems, now widely described as flawed and superficial. Based on this and similar comments, the Department has revised criterion (D)(2), now titled Improving Teacher and Principal Effectiveness Based on Performance, to encourage the design of high-quality evaluation systems, and to promote their use for feedback, professional improvement, and decision-making. The Department concurs with the many commenters who cautioned that teacher and principal ``effectiveness'' should not be based solely on student test scores. In this notice, ``effectiveness'' is defined as based on input from multiple measures, provided that student growth is a significant factor. In addition, this notice re-emphasizes that it is student growth--not raw student achievement data or proficiency levels--that is the ``significant factor'' to be considered in evaluating effectiveness. Finally, this notice expands and improves the four selection criteria that deal with teacher and principal professional development (criteria (B)(3), (C)(3)(ii), (D)(2)(iv)(a), and (D)(5)). It clarifies that professional collaboration and planning time, individualized professional development plans, training and support in the analysis and use of data, classroom observations with immediate feedback, and other activities are critical to supporting the development of teachers and principals. Specifically, criterion (D)(1) (proposed (C)(1)), concerning high- quality pathways for aspiring teachers and leaders, has been expanded. It now includes a new criterion (D)(1)(iii), under which States will be evaluated based on the extent to which they have in place a process for monitoring, evaluating, and identifying areas of teacher and principal shortage and for preparing teachers and principals to fill these areas of shortage. Criterion (D)(2) (proposed (C)(2)) has been revised to focus on the design and use of rigorous, transparent, and fair evaluation systems that provide regular feedback on performance to teachers and principals. This criterion also has been changed to clarify that the LEAs, not the States, should implement the teacher and principal effectiveness reforms under this criterion, and that the role of the States is to support their participating LEAs in implementing these reforms. Criterion (D)(2)(ii) (proposed (C)(2)(b)) now emphasizes that these evaluation systems should differentiate effectiveness using multiple rating categories, and should be designed and developed with teacher and principal involvement. Criterion (D)(2)(iii) (proposed criteria (C)(2)(c) and (C)(2)(d)(i)) encourages such evaluations to be conducted annually and to include timely and constructive feedback, while criterion (D)(2)(iv) (proposed criterion (C)(2)(d)) addresses uses of evaluations to inform decision-making. Criteria (D)(2)(iv)(c) and (D)(2)(iv)(d) (proposed criterion (C)(2)(d)(iii)) separately address the use of these evaluation systems to inform decisions regarding whether to grant tenure and/or full certification to effective teachers and principals (in criterion (D)(2)(iv)(c)), and removing ineffective teachers and principals (in criterion (D)(2)(iv)(d)). In addition, the Department has clarified that these decisions should be made using rigorous standards and streamlined, transparent, and fair procedures. Criterion (D)(3) (proposed (C)(3)) has been revised to clarify that the State's plan for the equitable distribution of effective teachers and principals should be informed by the State's prior actions and data, and should ensure that students in high-poverty as well as high- minority schools have equitable access to highly effective teachers and principals--and are not served by ineffective ones at higher rates than are other students. The performance measures for this criterion now include, for comparison purposes, data on the presence of highly effective and ineffective teachers and principals in low-poverty and low-minority schools. Criterion (D)(4) concerns improving the effectiveness of teacher and principal preparation programs. Criterion (D)(4)(i) (proposed (C)(4)) was revised to specify that, when reporting the effectiveness of teacher and principal credentialing programs, States should report student growth as well as student achievement data; they should report the data for all in-State credentialing programs, regardless of the number of graduates; and they should publicly report data, not ``findings.'' Criterion (D)(4)(ii) has been added to encourage States to expand those preparation and credentialing options and programs that are successful at producing effective teachers and principals (both as defined in this notice). Criterion (D)(5) (proposed criterion (C)(5)) focuses on providing effective support to teachers and principals. Here, the Department has inserted a new paragraph, (D)(5)(i), to provide additional guidance on, and examples of, effective support. The Department has also removed the reference to using [[Page 59691]] ``rapid-time'' student data to inform and guide the supports provided to teachers and principals. Turning Around the Lowest-Achieving Schools The Department made three noteworthy changes to the selection criteria on turning around the persistently lowest-achieving schools. First, this notice removes the restriction, proposed in the NPP, that permitted the ``transformation'' model to be used solely as a last resort. Instead, we simply specify that an LEA with more than nine persistently lowest-achieving schools may not use the transformation model for more than 50 percent of its schools. Second, the Department has fully aligned the school intervention requirements and definitions across Race to the Top, the State Fiscal Stabilization Fund, and the forthcoming Title I School Improvement Grants final notice. The Department's intention, in so doing, is to make it easier for States to develop consistent and coherent plans across these three programs. Third, the public comments suggested that there was confusion about the role of charter schools in the Department's reform agenda. Some commenters concluded that by placing the charter school criterion in the school turnaround section, the Department was advancing charter schools as the chief remedy for addressing the needs of the persistently lowest-achieving schools. While the Department believes that charter schools can be strong partners in school turnaround work, it does not believe that charter schools are the only or preferred solution to turning around struggling schools. In fact, it is the Department's belief that turning around the persistently lowest- achieving schools is a core competency that every district needs to develop, and that closing bad schools and opening good ones is the job of school district leaders. Notwithstanding research showing that charter schools on average perform similarly to traditional public schools, a growing body of evidence suggests that high-quality charter schools can be powerful forces for increasing student achievement, closing achievement gaps, and spurring educational innovation. As a consequence, the selection criterion pertaining to charter schools (criterion (F)(2), proposed (D)(2)) has been shifted from the Turning Around the Lowest-Achieving Schools section to the General section, where it more appropriately reflects charter schools' broader role as a tool for school innovation and reform. Specifically, the following changes have been made to criterion (E)(2) (proposed criterion (D)(3)), regarding turning around the lowest-achieving schools. Criterion (E)(2)(i) (proposed (D)(3)(i)) has been changed to allow States, at their discretion, to use Race to the Top funds to turn around non-Title I eligible secondary schools that would be considered ``persistently lowest-achieving schools'' if they were eligible to receive Title I funds. Criterion (E)(2)(ii) (proposed criterion (D)(3)(ii)) has been changed by removing the clause that restricted the use of the ``transformation'' model to situations where the other intervention models were not possible and by specifying that an LEA with more than nine persistently lowest-achieving schools may not use the transformation model for more than 50 percent of its schools. In addition, the four intervention models LEAs may use under this criterion are now described in detail in Appendix C, and these models have been made identical across the Race to the Top, State Fiscal Stabilization Fund, and Title I School Improvement Grants notices. Finally, the evidence collected for criterion (E)(2) will include the State's historic performance on school turnaround efforts, as evidenced by the total number of persistently lowest-achieving schools that States or LEAs attempted to turn around in the last five years, the approach used, and the results and lessons learned to date. General The General section includes a number of other key reform conditions or plans. First, criterion (F)(1) concerns education funding across the State. Criterion (F)(1)(i) (proposed (E)(2)) addresses the State's efforts to maintain education funding between FY 2008 and FY 2009. New criterion (F)(1)(ii) has been added to reward States whose policies lead to equitable funding between high-need LEAs and other LEAs, and within LEAs, between high-poverty schools and other schools. As noted above, criterion (F)(2) regarding charter schools has been moved to the General section from the Turning Around the Lowest- Achieving Schools section, where it was proposed criterion (D)(2). In this notice, the Department maintains its focus on high-quality charter schools as important tools for school reform. As was the case with the NPP, the final charter school criteria presented under (F)(2) encourage both unrestrictive charter school growth laws and strong charter school accountability. In support of charter school growth, the criteria also provide for the evaluation of States based on the extent to which they provide equitable funding for charter schools and offer them access to facilities. Criterion (F)(2)(ii) has also been revised to urge authorizers to encourage charter schools that serve student populations that are similar to local district student populations, especially relative to high-need students. In their comments, a number of States argued that they had laws-- other than charter school laws--that spurred school innovation. In response to these comments, the Department has added a new criterion, (F)(2)(v), that invites States to describe the extent to which they enable LEAs to operate innovative, autonomous public schools other than charter schools. It is the Department's hope that the Race to the Top competition gives States ample opportunity to explain and implement proven and promising ideas for bolstering student learning and educational attainment, and to do this in ways that work best in their local contexts. To ensure that the application reflects a broad range of effective State and local solutions, criterion (F)(3) (proposed criterion (E)(1)(iii)) asks States to describe laws, regulations, or policies (other than those asked about in other selection criteria) that have created conditions in the State that are conducive to education reform and improved student outcomes. Priorities Many commenters offered suggestions about the proposed priorities, in particular the invitational and competitive preference priorities. A number of commenters urged the Department to increase the importance of each invitational priority by making it a competitive or absolute priority, while others wanted to add new priorities. Because of the Department's desire to give States latitude and flexibility in developing focused plans to best meet their students' needs, we are not changing any of the priorities from invitational to competitive or absolute. We did, however, add a new invitational priority and make some changes to the proposed priorities. Regarding the proposed absolute priority, which stated that States' applications must comprehensively and coherently address all of the four education reform areas specified in the ARRA, the Department has added the requirement that States must comprehensively and coherently address the new State Success Factors criteria as well. [[Page 59692]] The final notice adds a new invitational priority 3, Innovations for Improving Early Learning Outcomes, expressing the Secretary's interest in applications that will improve early learning outcomes for high-need students who are young children. In invitational priority 4 (proposed priority 3), Expansion and Adaptation of Statewide Longitudinal Data Systems, programs such as at- risk and dropout prevention programs, school climate and culture programs, and early learning programs have been added to the list of programs that a State may choose to integrate with its statewide longitudinal data system. In invitational priority 5 (proposed priority 4), P-20 Coordination, Vertical and Horizontal Alignment, horizontal coordination of services was added as a critical component for supporting high-need students. In invitational priority 6 (proposed priority 5), School-level Conditions for Reform, Innovation, and Learning, new paragraph (vi) adds school climate and culture, and new paragraph (vii) adds family and community engagement to the list of school conditions conducive to reform and innovation. Requirements The first eligibility requirement, requirement (a), has been changed to provide that a State must have both phases of its State Fiscal Stabilization Fund application approved by the Department prior to being awarded a Race to the Top grant. In the NPP, we proposed that a State would have to receive approval of its Stabilization Fund applications prior to December 31, 2009 (for Race to the Top Phase 1 applicants) or prior to submitting a Race to the Top application (for Race to the Top Phase 2 applicants). The second eligibility requirement, requirement (b), was revised to clarify that the State must not have any legal, statutory, or regulatory barriers at the State level to linking data on student achievement (as defined in this notice) or student growth (as defined in this notice) to teachers and principals for the purpose of teacher and principal evaluation. In addition, several changes were made to the application requirements. The Department removed two proposed application requirements, application requirements (c) and (d), which would have required States to provide information about making education funding a priority and about stakeholder support. Note that the final notice retains the selection criteria that request this same information. Application requirement (c)(2) provides additional clarity about how to calculate the relative shares of the Race to the Top grant that participating LEAs will be eligible to receive. The Department has added a new application requirement, requirement (g), to clarify specific issues related to the term ``subgroup,'' to NAEP, and to the assessments required under the ESEA. In addition to requiring States to include, at a minimum, the listed student subgroups when reporting past outcomes and setting future targets, this application requirement includes statutory references. This addition eliminates the need for statutory references that define subgroups elsewhere in the notice, and they therefore have been removed. The program requirements have also changed. First, the Department has indicated its final approach to evaluation. The Institute of Education Sciences will conduct a series of national evaluations of Race to the Top State grantees as part of its evaluation of programs funded under the ARRA. States that are awarded Race to the Top grants will be required to participate in these evaluations and are welcome, but not required, to conduct their own independent, statewide evaluations as well. Finally, the program requirements have clarified that funds awarded under this competition may not be used to pay for costs related to statewide summative assessments. Definitions The Department has revised the definition of alternative routes to certification to require that in addition to the other program characteristics listed, the program must be selective in accepting candidates. The revised definition also clarifies that such programs should include standard features of high-quality preparation programs and award the same level of certification that is awarded by traditional preparation programs. A new definition of college enrollment refers to the enrollment of students who graduate from high school consistent with 34 CFR 200.19(b)(1) and who enroll in an institution of higher education (as defined in section 101 of the Higher Education Act, Public Law 105-244, 20 U.S.C. 1001) within 16 months of graduation. The final notice revises the definitions of effective teacher, effective principal, highly effective teacher, and highly effective principal to require that multiple measures be used to evaluate effectiveness, and provides several examples of appropriate measures. The definition of formative assessment has been revised to clarify that formative assessments are assessment questions, tools and processes and to require that feedback from such assessments need only be timely rather than instant. Under a new definition of high-minority school, States are to define high-minority schools in their applications in a manner consistent with their Teacher Equity Plans. The definition of high-need LEA was changed to conform with the definition of this term used in section 14013 of the ARRA. The final notice adds and defines high-need students to mean students at risk of educational failure or otherwise in need of special assistance and support, such as students who are living in poverty, who attend high-minority schools (as defined in this notice), who are far below grade level, who have left school before receiving a regular high school diploma, who are at risk of not graduating with a diploma on time, who are homeless, who are in foster care, who have been incarcerated, who have disabilities, or who are English language learners.\1\ --------------------------------------------------------------------------- \1\ The term English language learner, as used in this notice, is synonymous with the term limited English proficient, as defined in section 9101 of the ESEA. --------------------------------------------------------------------------- The final notice adds a definition of high-performing charter school. This definition refers to a charter school that has been in operation for at least three consecutive years and has demonstrated overall success, including substantial progress in improving student achievement and having the management and leadership necessary to overcome initial start-up problems and establish a thriving, financially viable charter school. The definition of high-quality assessment has been revised to clarify that test design must, to the extent feasible, use universal design principles in development and administration, and incorporate technology where appropriate. The final notice also adds a definition of increased learning time, which refers to using a longer school day, week, or year schedule to significantly increase the total number of school hours to include additional time for (a) instruction in core academic subjects, including English; reading or language arts; mathematics; science; foreign languages; civics and government; economics; arts; history; and geography; (b) instruction in other subjects and enrichment activities that contribute to a well-rounded education, including, for [[Page 59693]] example, physical education, service learning, and experiential and work-based learning opportunities that are provided by partnering, as appropriate, with other organizations; and (c) teachers to collaborate, plan, and engage in professional development within and across grades and subjects. The final notice adds a definition of innovative, autonomous public schools to refer to open enrollment public schools that, in return for increased accountability for student achievement (as defined in this notice), have the flexibility and authority to define their instructional models and associated curriculum; select and replace staff; implement new structures and formats for the school day or year; and control their budgets. In the definition of instructional improvement systems, the Department now provides examples of related types of data that could be integrated into these systems. The final notice adds a definition of involved LEAs, which refers to LEAs that choose to work with the State to implement those specific portions of the State's plan that necessitate full or nearly full statewide implementation, such as transitioning to a common set of K-12 standards, (as defined in this notice). Involved LEAs do not receive a share of the 50 percent of a State's grant award that it must subgrant to LEAs in accordance with section 14006(c) of the ARRA; however, States may provide other funding to involved LEAs under the State's Race to the Top grant in a manner that is consistent with the State's application. The final notice adds a definition of low-minority school, which is to be defined by the State in a manner consistent with the State's Teacher Equity Plan. A new definition of low-poverty school refers, consistent with section 1111(h)(1)(C)(viii) of the ESEA, to a school in the lowest quartile of schools in the State with respect to poverty level, using a measure of poverty determined by the State. The final notice adds a definition of participating LEAs, which refers to LEAs that choose to work with the State to implement all or significant portions of the State's Race to the Top plan, as specified in each LEA's agreement with the State. Each participating LEA that receives funding under Title I, Part A will receive a share of the 50 percent of a State's grant award that the State must subgrant to LEAs, based on the LEA's relative share of Title I, Part A allocations in the most recent year (that is, 2009), in accordance with section 14006(c) of the ARRA. Any participating LEA that does not receive funding under Title I, Part A (as well as one that does) may receive funding from the State's other 50 percent of the grant award, in accordance with the State's plan. The term persistently lowest-performing schools has been changed to persistently lowest-achieving schools. The definition has been revised to include the lowest-achieving five percent criterion originally included in proposed criterion (D)(3) and to add high schools with graduation rates below 60 percent. The definition also provides that, in determining the lowest-achieving schools, a State must consider the academic achievement of the ``all students'' group for each school in terms of proficiency on the State's assessments required by the ESEA in reading/language arts and mathematics combined, and the lack of progress by that group on these assessments over a number of years. The definition of rapid-time, in reference to reporting and availability of data, has been changed to remove the specification of a turnaround time of 72 hours and to clarify that it refers to locally collected school- and LEA-level data. The definition of student achievement has been revised to include several examples of alternate measures of student learning and performance for non-tested grades and subjects. The final notice also clarifies that, for tested grades and subjects, student achievement can be measured using alternative measures of student learning and performance in addition to the State's assessments under the ESEA. Finally, the reference to Individualized Education Program (IEP) goals as a potential achievement measure has been removed. The definition of student growth was clarified to mean the change in student achievement (as defined in this notice) for an individual student between two or more points in time, rather than just between two points in time, as the NPP had proposed, and that a State may also include other measures that are rigorous and comparable across classrooms. In the following section, the Department has summarized and provided its responses to the comments received. Analysis of Comments and Changes: An analysis of the comments and of any changes in the priorities, requirements, definitions, and selection criteria since publication of the NPP follows. General Comments on the Race to the Top Program Reorganization of the Final Notice Comment: None. Discussion: The selection criteria in this notice are reordered. The most significant change is the addition of State Success Factors to the beginning of the selection criteria. State Success Factors criteria include some new criteria, as well as criteria that are adapted from proposed criteria from the overall selection criteria section proposed in the NPP. This reorganization will give States the opportunity to begin their proposals with clear statements of their coherent and coordinated statewide reform agendas. However, with this change, it was necessary to redesignate the remaining criteria. For example, in the NPP, the criteria related to standards and assessments were designated as ``A'' (e.g., (A)(1), (A)(2), etc.), but in this final notice have been re-designated as ``B'' (e.g., (B)(1), (B)(2), etc.). One way to indicate this change throughout the final notice is to include both references every time a criterion is used (e.g., revised criterion (B)(1) (proposed criterion (A)(1)). Given the length of this notice and the extensive references to criteria, we have opted to refer only to the revised designation in the discussion of the comments. For example, we refer to a criterion for standards and assessments as ``criterion (B)(1),'' rather than as ``revised criterion (B)(1) (proposed criterion (A)(1)).'' In a few instances, we refer to ``proposed criterion'' or ``revised criterion'' for clarity but, generally, do not refer to each criterion with both its ``revised'' and ``proposed'' designation. We believe this format makes the document easier to read and understand. As a reminder to readers, we include both the final and proposed designations under the appropriate headings. Table 1 lists the final criteria and the corresponding proposed criteria. In Table 2, the columns are reversed to show the proposed criteria and the corresponding final criteria. There is a similar re-designation of the priorities. Specifically, we added a new invitational priority on innovations for improving early learning outcomes and designated it as priority 3. Subsequent priorities were re-numbered, and thus, proposed priorities 3, 4, and 5 are now priorities 4, 5, and 6, respectively. As with the selection criteria, generally, we will refer only to the final designation for these priorities and will use headers, as appropriate, to remind the reader of the changes. Thus, for example, we will refer to the priority [[Page 59694]] on Expansion and Adaptation of Statewide Longitudinal Data Systems, which was proposed priority 3 in the NPP, as priority 4. Table 3 summarizes these changes. Changes: We have re-designated the selection criteria and proposed priorities 3, 4, and 5. We will refer to the selection criteria and priorities with their final designations throughout this notice and, in a few instances, will refer to proposed designations for clarity. Three tables have been added to show how the final selection criteria and priorities relate to the proposed criteria and priorities. Table 1--The Final Selection Criteria Compared With the Proposed Selection Criteria ------------------------------------------------------------------------ Final notice Proposed notice ------------------------------------------------------------------------ A. State Success Factors (E)(1), (E)(3), (E)(4), (E)(5), and new A1. Articulating State's (E)(3)(iv), new education reform agenda and LEAs' participation in it (A)(1)(i) New (A)(1)(ii) (E)(3)(iv) (A)(1)(ii)(a) (E)(3)(iv) (A)(1)(ii)(b) (E)(3)(iv) (A)(1)(ii)(c) (E)(3)(iv) (A)(1)(iii)(a) (E)(3)(iv) and (E)(4)(i) (A)(1)(iii)(b) (E)(3)(iv) and (E)(4)(ii) (A)(1)(iii)(c) (E)(3)(iv) and (E)(4)(iii) (A)(1)(iii)(d) (E)(3)(iv) and new A2. Building strong statewide (E)(3)(i-ii), (E)(5), and new capacity to implement, scale up, and sustain proposed plans (A)(2)(i)(a) New (A)(2)(i)(b) (E)(5)(ii) (A)(2)(i)(c) (E)(5)(i) (A)(2)(i)(d) (E)(5)(v) (A)(2)(i)(e) (E)(5)(iii) (A)(2)(ii)(a) (E)(3)(i) (A)(2)(ii)(b) (E)(3)(i-ii) A3. Demonstrating significant (E)(1) and (E)(4) progress in raising achievement and closing gaps (A)(3)(i) (E)(1)(i-ii) (A)(3)(ii)(a) (E)(1)(iv) (A)(3)(ii)(b) (E)(1)(iv) (A)(3)(ii)(c) (E)(1)(iv) B. Standards and Assessments A. Standards and Assessments B1. Developing and adopting (A)(1) common standards (B)(1)(i)(a) (A)(1)(i) and (A)(1)(ii) (B)(1)(i)(b) (A)(1)(i) and (A)(1)(ii) (B)(1)(i)(c) (A)(1)(i) and (A)(1)(ii) (B)(1)(ii)(a) (A)(1)(i) (B)(1)(ii)(b) (A)(1)(ii) B2. Developing and implementing (A)(2) common, high-quality assessments (B)(2)(a) (A)(2) (B)(2)(a) (A)(2) B3. Supporting the transition (A)(3) to enhanced standards and high- quality assessments C. Data Systems to Support B. Data Systems to Support Instruction Instruction C1. Fully implementing a (B)(1) statewide longitudinal data system C2. Accessing and using State (B)(2) data C3. Using data to improve (B)(3) instruction (C)(3)(i) (B)(3)(i) (C)(3)(ii) New (C)(3)(iii) (B)(3)(ii) D. Great Teachers and Leaders C. Great Teachers and Leaders D1. Providing high-quality (C)(1) pathways for aspiring teachers and principals (D)(1)(i) (C)(1) (D)(1)(ii) (C)(1) (D)(1)(iii) New D2. Improving teacher and (C)(2) principal effectiveness based on performance (D)(2)(i) (C)(2)(a) (D)(2)(ii) (C)(2)(b) (D)(2)(iii) (C)(2)(c) and (C)(2)(d)(i) (D)(2)(iv) (C)(2)(d) (D)(2)(iv)(a) (C)(2)(d)(i) (D)(2)(iv)(b) (C)(2)(d)(ii) (D)(2)(iv)(c) (C)(2)(d)(iii) (D)(2)(iv)(d) (C)(2)(d)(iii) D3. Ensuring equitable (C)(3) distribution of effective teachers and principals [[Page 59695]] (D)(3)(i) (C)(3) (D)(3)(ii) (C)(3) D4. Improving the effectiveness (C)(4) of teacher and principal preparation programs (D)(4)(i) (C)(4) (D)(4)(ii) New D5. Providing effective support (C)(5) to teachers and principals (D)(5)(i) (C)(5) (D)(5)(ii) (C)(5) E. Turning Around the Lowest- D. Turning Around Struggling Achieving Schools Schools E1. Intervening in the lowest- (D)(1) achieving schools and LEAs E2. Turning around the lowest- (D)(3) achieving schools (E)(2)(i) (D)(3)(i) (E)(2)(ii) (D)(3)(ii) F. General Selection Criteria (D)(2), (E)(1), (E)(2), and new F1. Making education funding a (E)(2) and new priority (F)(1)(i) (E)(2) (F)(1)(ii) New F2. Ensuring successful (D)(2) conditions for high-performing charter schools and other innovative schools (F)(2)(i) (D)(2)(i) (F)(2)(ii) (D)(2)(ii) (F)(2)(iii) (D)(2)(iii) (F)(2)(iv) (D)(2)(iv) (F)(2)(v) New F3. Demonstrating other (E)(1)(iii) significant reform conditions Removed (E)(3)(iii) Removed (E)(5)(iv) ------------------------------------------------------------------------ Table 2--The Proposed Selection Criteria Compared With the Final Selection Criteria ------------------------------------------------------------------------ Proposed notice Final notice ------------------------------------------------------------------------ A. Standards and Assessments B. Standards and Assessments (A)(1). Developing and adopting (B)(1) common standards (A)(1)(i) (B)(1)(i), (B)(1)(ii)(a) (A)(1)(ii) (B)(1)(i), (B)(1)(ii)(b) (A)(2). Developing and (B)(2) implementing common, high- quality assessments (A)(3). Supporting the (B)(3) transition to enhanced standards and high-quality assessments B. Data Systems to Support C. Data Systems to Support Instruction Instruction (B)(1). Fully implementing a (C)(1) statewide longitudinal data system (B)(2). Accessing and using (C)(2) State data (B)(3). Using data to improve (C)(3)(i), (C)(3)(iii) instruction (B)(3)(i) (C)(3)(i) (B)(3)(ii) (C)(3)(iii) C. Great Teachers and Leaders D. Great Teachers and Leaders (C)(1). Providing high-quality (D)(1)(i-ii) pathways for aspiring teachers and principals (C)(2). Improving teacher and (D)(2) principal effectiveness based on performance (C)(2)(a) (D)(2)(i) (C)(2)(b) (D)(2)(ii) (C)(2)(c) (D)(2)(iii) (C)(2)(d)(i) (D)(2)(iii), (D)(2)(iv)(a) (C)(2)(d)(ii) (D)(2)(iv)(b) (C)(2)(d)(iii) (D)(2)(iv)(c), (D)(2)(iv)(d) (C)(3). Ensuring equitable (D)(3)(i), (D)(3)(ii) distribution of effective teachers and principals (C)(4). Reporting the (D)(4)(i) effectiveness of teacher and principal preparation programs (C)(5). Providing effective (D)(5)(i), (D)(5)(ii) support to teachers and principals D. Turning Around Struggling E. Turning Around the Lowest- Schools Achieving Schools (D)(1). Intervening in the (E)(1) lowest-achieving schools and LEAs (D)(2). Increasing the supply (F)(2) of high-quality charter schools (D)(2)(i) (F)(2)(i) (D)(2)(ii) (F)(2)(ii) (D)(2)(iii) (F)(2)(iii) (D)(2)(iv) (F)(2)(iv) (D)(3). Turning around the (E)(2) lowest-achieving schools [[Page 59696]] (D)(3)(i) (E)(2)(i) (D)(3)(ii) (E)(2)(ii) E. Overall Selection Criteria (A) State Success Factors and (F) General Selection Criteria (E)(1). Demonstrating (A)(3)(i), (A)(3)(ii), (F)(3) significant progress (E)(1)(i) (A)(3)(i) (E)(1)(ii) (A)(3)(i) (E)(1)(iii) (F)(3) (E)(1)(iv) (A)(3)(ii) (E)(2). Making education (F)(1)(i) funding a priority (E)(3). Enlisting statewide (A)(1)(ii), (A)(1)(iii), support and commitment (A)(2)(ii) (E)(3)(i) (A)(2)(ii)(a), (A)(2)(ii)(b) (E)(3)(ii) (A)(2)(ii)(b) (E)(3)(iii) Removed (E)(3)(iv) (A)(1)(ii), (A)(1)(iii) (E)(4). Raising achievement and (A)(1)(iii) closing gaps (E)(4)(i) (A)(1)(iii)(a) (E)(4)(ii) (A)(1)(iii)(b) (E)(4)(iii) (A)(1)(iii)(c) (E)(5). Building strong (A)(2)(i)(b-e) statewide capacity to implement, scale up, and sustain proposed plans (E)(5)(i) (A)(2)(i)(c) (E)(5)(ii) (A)(2)(i)(b) (E)(5)(iii) (A)(2)(i)(e) (E)(5)(iv) Removed (E)(5)(v) (A)(2)(i)(d) New (A)(1)(i) New (A)(1)(iii)(d) New (A)(2)(i)(a) New (C)(3)(ii) New (D)(1)(iii) New (D)(4)(ii) New (F)(1)(ii) New (F)(2)(v) ------------------------------------------------------------------------ Table 3--The Final Priorities Compared With the Proposed Priorities ------------------------------------------------------------------------ Final priorities Proposed priorities ------------------------------------------------------------------------ Priority 1: Absolute Priority-- Priority 1: Absolute Priority. Comprehensive Approach to Education Reform. Priority 2: Competitive Preference Priority 2: Competitive Preference Priority--Emphasis on Science, Priority. Technology, Engineering, and Mathematics (STEM). Priority 3: Invitational Priority-- New. Innovations for Improving Early Learning Outcomes. Priority 4: Invitational Priority-- Priority 3. Expansion and Adaptation of Statewide Longitudinal Data Systems. Priority 5: Invitational Priority-- Priority 4. P-20 Coordination, Vertical and Horizontal Alignment. Priority 6: Invitational Priority-- Priority 5. School-Level Conditions for Reform, Innovation, and Learning. Priority 6, Paragraph vi. New. Priority 6, Paragraph vii. New. ------------------------------------------------------------------------ Overall Comments on the Race to the Top Program Comment: We received a number of comments that addressed issues related to the Race to the Top program in general, as well as comments that focused on a number of priorities and selection criteria. Discussion: We are addressing, in this section, general comments on the Race to the Top program, as well as comments that focused on multiple priorities and selection criteria. This allows us to group similar comments and be more responsive to the commenters. Changes: None. Comment: Many commenters supported our proposals in the NPP and our effort to leverage cutting-edge education reforms and innovation in a competitive Race to the Top program that will lay the foundation for significant improvement of America's education system. In particular, these commenters praised the Department's proposals for ``game- changing'' reforms in the areas of improving teacher and principal effectiveness and turning around our lowest-achieving schools. Other commenters expressed their overall opposition to the Race to the Top program because of what they described as its ``one-size-fits- all'' approach to education reform involving ``a top-down, narrow definition of innovation that has little research to support it.'' Another commenter stated that the Department is prescribing a national formula for education reform, which threatens to undermine the program. In particular, several [[Page 59697]] commenters objected to the proposed use of test scores as an accurate measure of student achievement and what they claimed were ``unproven'' interventions such as charter schools and linking teacher compensation to student achievement data. Many commenters asserted that the proposed program design would interfere with State and local prerogatives and responsibilities for public education. Other commenters noted that some of the interventions proposed in Race to the Top, such as increasing the number of high-quality charter schools, are not consistent with existing State laws and might not work as well in rural areas as in urban environments. One commenter stated that the NPP ignored the existing ESEA school improvement process and ``would simply layer another top-down accountability process on top of the current faulty one.'' Some of these commenters urged that the final notice instead encourage States to propose multiple innovative, research-based reform strategies and models tailored to their own unique local needs. Discussion: The Department appreciates the expressions of support for its Race to the Top proposal as well as commenters' constructive suggestions. The Race to the Top program provides a flexible framework for comprehensive State and local innovation in the key reform areas identified in the ARRA. In fact, one of the key purposes of this program is to ask States for their best ideas about how to address the levers of change--the four assurances in the ARRA--to significantly improve student outcomes and advance the field of education reform. To create ``room'' for States to meet this goal, this final notice, consistent with the NPP, includes only one absolute priority and two eligibility requirements--none of which interferes with a State's flexibility to put forward its best ideas and practices for reform. The absolute priority focuses on comprehensiveness and coherence across the reform areas, and the eligibility requirements include (1) approved applications for funding under Phase 1 and Phase 2 of the Stabilization program, and (2) no legal, statutory, or regulatory barriers at the State level to linking data on student achievement or student growth to teachers and principals for the purpose of teacher and principal evaluation. As we noted in the NPP, section 14005(d) of the ARRA requires a State that receives funds under the Stabilization program to provide assurances in the same four education reform areas that are advanced by the Race to the Top program. We, therefore, believe it would be inconsistent to award a Race to the Top grant, which requires a determination that a State has made significant progress in the four education reform areas, to a State that has not met requirements for receiving funds under the Stabilization program. With regard to the second eligibility requirement, we believe that the capability to link student achievement to teachers and principals for the purposes of evaluation is fundamental to the Race to the Top reforms and to the requirement in section 14005(d)(2) of the ARRA that States take actions to improve teacher effectiveness. Furthermore, without the legal authority to use student achievement or student growth data for teacher and principal evaluations, States would not be able to execute reform plans related to several selection criteria in this notice. In addition, the proposed selection criteria were not designated as eligibility requirements; instead, they were proposed as recommended elements of a comprehensive State plan that would provide an individual State with the flexibility to emphasize its own priorities and craft a winning application. This flexible approach has been retained in this final notice. For instance, States need not address every selection criterion, so long as they comprehensively and coherently address all of the four education reform areas as well as the State Success Factors Criteria. Through this program, the Department will reward success in at least two ways: First, by giving States credit for having already put into place key conditions for reform, improving student achievement, and closing achievement gaps; and second, by encouraging States to build on their assets and successes. We believe that State plans that build on a foundation of successful existing practices will be more likely to succeed in improving student outcomes. It is important to note that the Race to the Top program is a voluntary competitive grant program. Consistent with section 14006(b) of the ARRA, we may use ``such other criteria as the Secretary determines appropriate'' in making Race to the Top awards; our intention is not to fund every State but to identify and reward the subset of States that demonstrate the greatest promise of making meaningful gains in developing standards and assessments, using data to drive improved student outcomes, improving teacher and principal effectiveness and achieving equity in the distribution of effective teachers and principals, and turning around struggling schools. Moreover, because the effects of the Race to the Top program might not be captured by existing State accountability systems, such as those created under the ESEA, this final notice retains the separate performance measures included in the NPP. In response to commenters' concerns pertaining to ``unproven'' interventions in the Race to the Top program, there is ample evidence, for example, that high-performing charter schools can significantly improve the achievement of high-need students. Likewise, the research supports that effective teachers and principals are essential to improving student achievement; accordingly, the Department believes that identifying, recruiting, developing, and retaining effective teachers and school leaders is critical to creating high-performing schools and a world-class education system. Finally, we are providing States with flexibility to incorporate these reforms into their plans through their own innovative and thoughtful approaches that are designed to address their specific needs. In addition, we are including in this final notice two additional criteria intended to make this flexibility for innovation more explicit. Changes: We have added the following criteria: First, criterion (F)(2)(v) asks a State to demonstrate the extent to which it enables LEAs to operate innovative, autonomous public schools other than charter schools. Second, criterion (F)(3) (proposed criterion (E)(1)(iii)) encourages States to describe any other conditions favorable to education reform or innovation that have increased student achievement or graduation rates, narrowed achievement gaps, or resulted in other important outcomes. Transparency Comment: Some commenters requested that the Department make all State applications and annual reports publicly available for review. Additional commenters requested that applications and all related materials be posted online prior to approval. Discussion: To foster transparency and openness, the Department plans to post all State applications--for both successful and unsuccessful applications--on our Web site at the conclusion of each phase of the competition, together with the final scores each received. States may choose to make their applications publicly available at any time. We also anticipate making State annual reports publicly available. Changes: None. [[Page 59698]] Allocation of Points Comment: Several commenters requested clarification concerning the weighting of selection criteria. Two commenters specifically requested that the point scale or rubric be disclosed. Other commenters suggested that the point allocations be subject to public comment. One commenter suggested that Secretary Duncan make the final award selections. Discussion: To ensure that the Race to the Top competition is as open and transparent as possible, the Department is publishing the reviewer scoring rubric in Appendix B of this notice. The rubric is designed to ensure consistency across reviewers and help applicants better understand the Department's priorities for this competition by clearly identifying the point allocations for each selection criterion and indicating how priorities will be judged. The Secretary will select the grantees after considering the rank order of applications, each applicant's status with respect to the Absolute Priority and eligibility requirement (a), and any other relevant information. Grant award decisions are made by the Secretary, pursuant to the Department's regulations. It is the Department's practice to first take public comment on proposed selection criteria before making final decisions on those criteria. This allows the Department to consider public comment on the proposed selection criteria before making final decisions on point allocations, which are then published in the application package and final notice inviting applications. Changes: The scoring rubric for the criteria is included as Appendix B. Comment: Many commenters recommended weighting State Reform Conditions Criteria more heavily than Reform Plan Criteria, arguing that States that have already enacted reform policies are more likely to accelerate student achievement. On the other hand, one commenter suggested that States be given extra credit for recently enacted regulatory or legislative reforms, particularly in Phase 2 of the Race to the Top competition. Several other commenters recommended that the Department ensure that no single criterion or assurance, by itself, operate to eliminate a State from the Race to the Top competition. One of these commenters argued that States need flexibility, while another commenter added that a State application that addresses some criteria in depth may be stronger than one that addresses all criteria but is ``shallow'' in its overall approach. Discussion: The scoring rubric assigns more weight to accomplishments (i.e., State Reform Conditions Criteria) than to plans (i.e., Reform Plan Criteria). (See Appendix B). However, the Department will not give ``extra credit'' to States that have recently enacted laws or polices intended to support their Race to the Top applications, as that would penalize early reformers. Finally, as is made clear elsewhere in this notice, the selection criteria are not eligibility requirements; the failure to meet any single criterion, or even a number of criteria, will not preclude a State from receiving a Race to the Top award. Moreover, the large number of criteria for which a State may earn points means that an application that is exceptionally strong on a majority of, but not all, Race to the Top selection criteria may score higher than an application that earns only partial credit on every criterion. On the other hand, applicants should keep in mind the statutory emphasis on comprehensive reforms, as well as absolute priority 1, which requires an applicant to address comprehensively all four ARRA assurance areas as well as the State Success Factors (Section (A)) of the selection criteria. Changes: None. Comment: Many commenters recommended that the Department heavily weight the selection criteria for turning around struggling schools. Another commenter suggested a weighting system that rewards States for providing flexibility or autonomy to schools, whether charter or traditional. One commenter suggested awarding a significant portion of points for activities that support science, technology, engineering, and mathematics (STEM) initiatives; needy locations; turning around school climate; partnerships with community based organizations and volunteers; and family engagement. Discussion: The Department believes that each of the four reform areas is critical and has assigned points accordingly. The Department, therefore, declines to heavily weight the selection criteria for turning around struggling schools or to provide extra points to States that provide flexibility and autonomy to its schools. We decline to award a significant portion of points for activities that support STEM initiatives, needy locations, school climate, partnerships with community based organizations and volunteers, and family engagement. We note that each of these areas already is addressed in this notice. For example, a State that includes STEM education in its comprehensive plan will be eligible to receive competitive preference points; States are required to give priority to high-need LEAs in their Race to the Top plans; and strategies to improve school climate, develop partnerships with CBOs, and improve family engagement are specifically encouraged in the school intervention models in Appendix C. Changes: None. Comment: One commenter suggested that the Department release guidance to help States determine whether they are likely to be successful in competing for Race to the Top funds as judged by their NAEP scores. The commenter suggested that States with low NAEP scores are unlikely to receive funds and would be wasting tremendous resources in completing a Race to the Top application. Discussion: The Department has created a scoring rubric with the number of possible points for each selection criterion. The rubric will be used by reviewers to judge State applications for Race to the Top funds. The Department is including the rubric in Appendix B to ensure that the scoring of State applications is transparent and so that States are fully informed as they develop their applications. We note that the criterion referenced by the commenter (proposed criterion (E)(1)(iv), which has been revised and designated as criterion (A)(1)(iii)), focuses on improvements in achievement, and not simply whether a State has high or low scores, as reported by both the NAEP and the assessments required under the ESEA. Changes: None. Other Education Reform Strategies Comment: Many commenters suggested that Race to the Top take into account existing State and local education reform strategies, particularly in high-need schools. Several commenters suggested that Race to the Top include reform initiatives specifically targeted to high schools, the learning needs of advanced students, or the attainment of ``21st Century Skills'' (described in the comments as skills pertaining to media, technology, and financial literacy and global awareness). One commenter urged a greater focus in Race to the Top on ``disruptive innovations'' such as online learning, while others championed specific subjects, such as music and the arts, as essential ways of engaging students in learning and keeping them in school. In addition, several commenters argued that the study of foreign languages is critical for our future competitiveness in the global economy and should be included as a priority in the Race to the Top program. [[Page 59699]] Discussion: The Department recognizes that numerous strategies, interventions, technologies, and subjects can make meaningful contributions to improving the quality of our education system, engaging students, and turning around the lowest-achieving schools. We also agree that it is important to give States credit for existing reforms that are achieving positive outcomes. This is one reason why we are clarifying and expanding criterion (F)(3) (proposed criterion (E)(1)(iii)) which, as mentioned earlier, asks States to demonstrate the extent to which they have created conditions favorable to education reform or innovation, in addition to the information provided under other State Reform Conditions Criteria. We also note that under the State Reform Conditions Criteria, States will be rewarded for having put into place key conditions for reform, while the State Reform Plan Criteria asks States to create plans that build on their successes. Changes: Criterion (F)(3) (proposed criterion (E)(1)(iii)) has been clarified and expanded to focus on the extent to which a State, in addition to information provided under other State Reform Conditions Criteria, has created, through law, regulation, or policy, other conditions favorable to education reform or innovation that have increased student achievement or graduation rates, narrowed achievement gaps, or resulted in other important outcomes. Evidence-Based Practices in Race to the Top Comment: Some commenters argued that the Race to the Top program, as outlined in the NPP, would not adequately support evidence-based practices. One of these commenters suggested including a minimum evidence threshold for each of the State Reform Plan Criteria. Discussion: We believe that the use of evidence-based practices is critical to the success of the Race to the Top program. However, we acknowledge that the research evidence to support education practices, strategies, and programs may not reach the same threshold for each reform area. The four education reform areas in the ARRA are in large part focused on giving educators new data-based tools for developing and implementing their own best practices. Indeed, developing stronger standards and assessments, expanding the use of longitudinal data systems, improving teacher and principal effectiveness, and supporting struggling schools are all intended to create and support evidence- driven continuous instructional improvement based on what works in the classroom. One key purpose of Race to the Top is to empower cutting- edge States and LEAs to build on what works while also creating new, more effective models of educational reform and improvement that will significantly expand our collection of evidence-based practices. We believe that State flexibility is key in this effort. Changes: None. Support for Traditional Public Schools Comment: One commenter claimed that the Race to the Top program, as outlined in the NPP, would result in little or no support for traditional public schools because it seemed primarily concerned with creating ``financial opportunities for educational entrepreneurs.'' Discussion: This commenter misconstrues Race to the Top, which is focused almost entirely on improving our traditional public schools. Furthermore, pursuant to section 14006(c) of the ARRA, at least 50 percent of Race to the Top funds must be allocated directly to LEAs according to their relative shares of funding under Title I, Part A of the ESEA; a majority of those LEAs are likely to serve exclusively traditional public schools. Further, each of the four assurances under the ARRA, which provide the overall framework for the Race to the Top program, is aimed at increasing the effectiveness of State and local support for traditional public schools. Changes: None. Eligibility of Other Entities Comment: Several commenters suggested that entities other than States be made eligible to apply directly for Race to the Top funds. Specifically, commenters suggested that such organizations as charter schools, independent school districts, community colleges, historically black colleges and universities, LEAs, and not-for-profit organizations partnering with either LEAs or universities be able to apply for Race to the Top funds. Those commenters argued that preventing these entities from applying for the Race to the Top competition would limit the creation of innovative partnerships. Other commenters requested that private schools and non-profit organizations that partner with LEAs be eligible. Another commenter suggested that municipalities, in addition to LEAs, should be eligible to receive Race to the Top subgrants. One commenter was supportive of States applying directly for funds as opposed to LEAs. Discussion: Section 14006(a)(2) of the ARRA specifically states that ``the Secretary shall make grants to States that have made significant progress'' in meeting the objectives of the four reform areas. As such, the Department does not have the authority to expand the statute's directive to extend eligibility to the other entities suggested by the commenters. The Department recognizes, however, that these entities and others within the State are essential to the success of Race to the Top grantees. For this reason, we are adding additional examples of stakeholders to State Success Factors Criterion (A)(2)(ii)(b) (proposed criteria (E)(3)(i) and (ii)), which specifically asks applicants to demonstrate the extent to which they have secured broad stakeholder support. In addition, participating LEAs may use their funds to serve non-Title I schools, if doing so aligns with the State's plan and the Department's general regulations on uses of funds. States also may, consistent with applicable procurement requirements, contract with organizations such as those mentioned by the commenters, using the State's share of Race to the Top funds. Changes: Criterion (A)(2)(ii)(b) has been expanded to include additional examples of stakeholder support. Comment: Some commenters suggested that private schools be eligible for Race to the Top funds. One commenter argued that services to students and teachers in private schools is permitted under the Stabilization Fund and, therefore, should be permitted under the Race to the Top program. The commenter stated that section 14006(b) of the ARRA leaves considerable discretion to the Secretary in awarding grants on the basis of State applications for the Stabilization Fund and argued that this latitude extends to Race to the Top funds. The commenter requested that the overall selection criteria be amended to include a criterion that focuses on applicants' compliance with statutory provisions related to the equitable participation of private school students and teachers in Federal education programs. Other commenters recommended that the notice encourage States to include faith-based schools in their applications. These commenters pointed to positive effects on at-risk youth attributed to Catholic and other faith-based schools. A few commenters specifically requested that faith-based schools be eligible to apply for Race to the Top funds directly. One commenter noted that because private school students participate in Title I, Part A programs under the ESEA, they should be allowed [[Page 59700]] to participate in the Race to the Top activities approved in a State's plan. Other commenters requested that private schools that partner with LEAs be made eligible to receive Race to the Top funds. One commenter asserted that private schools should have the option to participate in all Federal programs without sacrificing control in such areas as curriculum, hiring, or teacher requirements. Discussion: As described in the response to the previous set of comments related to eligibility, the statutory language of the ARRA specifically provides that States are the eligible applicants for Race to the Top funds, and that only LEAs are eligible to receive subgrants from the States. Race to the Top funds may not be provided to private schools through a grant or subgrant, and there is no requirement that private school students, teachers, or other educational personnel participate in Race to the Top on an equitable basis (as required in some programs in the ESEA). Furthermore, Race to the Top funds may not be used to provide financial assistance to students to attend private schools. However, States and LEAs have the flexibility to include private school students, teachers, and other educational personnel in activities that the States and LEAs deem appropriate, and may contract with private schools for appropriate secular activities, consistent with the State's plan. Changes: None. Authority for the NPP Comment: Some commenters objected to the NPP, arguing that it proposed education policy outside of the legislative process. One commenter claimed that while the ARRA ``imposes only brief and general requirements'' governing the use of Race to the Top funds, the prescriptive proposals in the NPP ``amount to writing new laws.'' One commenter recommended that Congress hold hearings on the notice, claiming that there has been a lack of sufficient time to review the NPP. Another commenter asserted that Congress should conduct a broad review of the NPP and of our determination that the NPP would ``not unduly interfere with State, local, and Tribal governments in the exercise of governmental functions.'' Two commenters also stated that it appeared that we were using Race to the Top, in the context of the fiscal emergency currently faced by many States, to impose education reform policies that would not otherwise be accepted by States and LEAs. Discussion: The commenters are correct that the ARRA offers few specifics governing the Race to the Top program; however, the ARRA is very clear that (1) The program is expected to provide incentives for ``significant progress'' in the four assurance areas, and (2) the Secretary has authority to award Race to the Top funds using ``such criteria as the Secretary determines appropriate.'' Moreover, section 410 of the General Education Provisions Act (20 U.S.C. 1221e-3) gives the Secretary full authority to promulgate rules and regulations necessary for the effective administration of Federal education programs. This final notice, like the NPP, is consistent with these authorities. Moreover, the ARRA specifically provides that Race to the Top funds must be awarded not later than September 30, 2010. In order to provide States the maximum amount of time possible to plan, organize, and draft their applications for the Phase 1 and Phase 2 competitions, while still allowing and responding fully to public comment, the Department sought comment on the NPP for a 30-day time period. Notably, section 437(d)(1) of the General Education Provisions Act, 20 U.S.C. 1232(d)(1), allows the Department to waive rulemaking for the first grant competition under a new or substantially revised program authority. The Race to the Top program is a new program, so the Department was not required to conduct notice-and-comment rulemaking. The Department, however, instead of taking advantage of that option, specifically sought public comment in order to inform the development of the program. Moreover, the comments received from over 1,100 commenters during the NPP's 30-day comment period suggest that this period of time was sufficient for broad public review and comment. In response to claims that the Race to the Top requirements would interfere with State, local, or Tribal governments or impose policies on these governments, we note that the Race to the Top program is a voluntary competitive grant program that, like other such programs, includes requirements and criteria that must be met in order for States to participate and receive funding. States and LEAs that do not wish to comply with these conditions and criteria are not required to apply for a grant. While the fiscal crises currently faced by many States may encourage States to apply for Race to the Top funds, ameliorating State and local deficits is not the primary purpose of this program. Instead, the Race to the Top program, which will award only about 4 percent of all education funds provided by the ARRA, was specifically intended to encourage and reward those States that are making ``significant progress'' toward the four assurances. This final notice, like the NPP, represents our effort to establish reasonable and appropriate criteria for defining the ``significant progress'' as required by the statute. Changes: None. Promoting Successful Implementation Comment: Several commenters raised questions concerning the implementation of Race to the Top. One commenter expressed concern that the proposed priorities pertained to State rather than LEA functions. The commenter noted that States do not achieve significant improvements in student outcomes; rather teachers working in LEAs with students, parents, school administrators, and other stakeholders make the difference. Another commenter urged us to make Race to the Top awards as soon as possible, but not later than early 2010, so that States and school districts can begin implementing reforms in the 2010-2011 school year. Two commenters suggested that we will not be able to create the momentum to accomplish national education reform unless a sufficient number of States receive Race to the Top funds. One commenter suggested that the Race to the Top program would have a broader national impact if 26-30 States participated in the program, and recommended structuring the award phases so that the first round provides large ``lead'' grant awards followed by a second round of smaller ``but still substantial'' awards. Discussion: The Department agrees that the success of a State's Race to the Top reform efforts will depend on its ability to articulate a comprehensive and coherent education reform agenda, secure the commitment of its LEAs to implement on its proposed plans, and provide leadership and support to its LEAs. We recognize that the most important reform efforts will take place in the classroom and that a critical part of a State's application will be the State's capacity to support its LEAs in successfully implementing its plans through such activities as identifying best practices, widely disseminating and replicating effective practices statewide, and holding LEAs accountable for progress and performance. We are aware of the need for successful applicants to begin implementing their Race to the Top plans as soon as possible. Toward this [[Page 59701]] end, we expect to make Phase 1 Race to the Top awards in the first half of calendar year 2010. We do not agree that Race to the Top funds should be spread across an arbitrarily larger number of States. Instead, the size and number of Race to the Top awards in the two phases of funding will depend on the scope and quality of the applications that States submit to the Department. Changes: None. Comment: Several commenters requested clarification regarding how States should develop and use performance and data indicators. One commenter suggested requiring States to provide information on the extent to which LEAs in the State have made adequate yearly progress (AYP) as part of their annual reports. Other commenters called for the Department to peer-review annual State Race to the Top reports. Two commenters expressed concern that performance measures would vary from State to State, causing confusion in the field. Finally, one commenter recommended that the Department remove the phrase ``ambitious yet achievable'' because its meaning is unclear. Discussion: In the NPP the Department proposed core performance measures for evaluating the performance of States receiving Race to the Top funds against both the four assurances and specific elements of State Race to the Top plans (see Appendix A). For the most part, we are retaining these measures, with some modifications, in this notice. The Department understands the concerns expressed by commenters about comparability of data across States receiving Race to the Top grants; this is one reason that this final notice retains the request for States to set student achievement and gap-closing goals based on NAEP data in revised criterion (A)(1)(iii) (proposed criterion (E)(4)). NAEP scores are comparable across States, thus eliminating concerns about the widely varying standards and assessments in use by States under ESEA accountability systems. States already issue annual reports on AYP status for schools and LEAs, including proficiency rates for all schools; there is no need to duplicate this reporting by requiring its inclusion in a State's annual Race to the Top report. However, States that desire to include AYP data in their annual Race to the Top reports are free to do so. The Department declines to add a requirement for peer review of these annual reports. Finally, we are retaining the ``ambitious yet achievable'' language throughout the Race to the Top State Reform Plan Criteria. As noted elsewhere in this notice, the Department believes that this language strikes the right balance between encouraging States to set a high bar for Race to the Top goals while recognizing that real change in education is difficult and takes time. The goal is to encourage realistic thinking and planning that connects specific activities to specific, achievable results, while acknowledging that improvements in the Nation's education system are urgently needed and the country's children cannot wait. Changes: None. Comment: One commenter expressed concern that too many of the measures proposed in the NPP reflect past performance and recommended a greater emphasis on future Race to the Top performance. Discussion: The emphasis on past performance comes directly from the requirements in the ARRA, which requires States to have made significant progress in the four education reform areas in order to receive a grant. Once Race to the Top grants are awarded and winning States begin implementing their reform plans, the Department will become far more focused on how States perform under this program. Changes: None. Race to the Top Funding Comment: Several commenters suggested that the Department provide more information on expected funding levels for States that receive Race to the Top funds, including the number and size of Race to the Top awards for both the Phase 1 and 2 competitions. Multiple commenters suggested that we provide funding for States to develop reform plans and applications. One commenter requested assurances that the level of funding to successful State applicants will be sufficient to carry out all activities in States' reform plans. Two commenters expressed concern that LEAs will have control of ARRA funds, outside of public accountability and without provisions for oversight, while another commenter requested information about the restrictions on the usage of Race to the Top funds, and an explanation of how States are expected to use them. Discussion: We encourage States to develop budgets that match the needs they have outlined in their applications. To support States in planning their budgets, we have developed nonbinding budget guidance with ranges for each State; these are listed in the notice inviting applications, published elsewhere in this issue of the Federal Register. These ranges may be used to guide States as they draft their applications, but States may prepare budgets that are above or below the suggested ranges. The amount awarded in Phase 1 will depend on the quality of the applications that States submit to the Department, as well as the successful applicants' proposed budgets. It is our intention to have significant funds available for Phase 2 applicants and awards. The ARRA does not provide funding to help States prepare or design their Race to the Top applications. Finally, the Department has taken extraordinary measures to ensure accountability in the use of all ARRA funds, including the Race to the Top fund, so that all dollars are used wisely and accounted for in a transparent manner. Indeed, as explained in the Reporting section of this final notice and in the notice inviting applications, successful applicants must comply with the ARRA annual reporting requirements in section 14008 of the ARRA and with quarterly reporting requirements in section 1512(c) of the ARRA, which are designed to ensure thorough and public oversight of the expenditure of ARRA funds. In addition, the Department has established a Recovery Act Web site and hotline for members of the public to report suspected misuse of funds. Changes: None. Comment: One commenter expressed concern about structuring the Race to the Top program as a competitive grant. The commenter noted that structuring the program this way will mean that not every State will win Race to the Top grant funds. Another commenter stated that by predetermining ``the conditions necessary for reform,'' the winners and losers have already been chosen. Discussion: The Race to the Top program is intended to promote and reward States making the most progress in achieving the goals described in the ARRA and by the Secretary. As the Secretary and the President have said, Race to the Top is designed as a competitive, once-in-a- lifetime opportunity for the Federal Government to create incentives for far-reaching improvement in our Nation's schools. While other ARRA funds provide substantial increases in formula funds to States (e.g., the Stabilization Fund, ESEA Title I, IDEA), we strongly believe that the competitive nature of the Race to the Top program will encourage statewide reform resulting in significant improvement in student outcomes. Finally, we note that contrary to the suggestion made by one commenter, the Department has not pre-selected the winners and losers for this competition. Applications will be judged based on the conditions States have put in place [[Page 59702]] by the time they apply, the strength of their plans, and how these come together as a coherent and cohesive strategy to improve student outcomes. Changes: None. Flexibility to Allocate Funds Comment: Several commenters sought greater flexibility for States and LEAs to determine award levels. For example, a few commenters suggested that allocating 50 percent of Race to the Top funds by formula runs counter to the program's goals, and that States should be allowed to focus funding on LEAs with the greatest need for additional resources to address the educational needs of at-risk students such as English language learners, students with disabilities, and students from low-income families, or to give priority to one or more of the four assurances when funding LEAs. Other commenters sought clarification about State flexibility in using the 50 percent of funds that will not be distributed on the basis of the Title I formula. One commenter suggested that States might use their shares of Race to the Top awards to support high-need students in non-Title I schools, while another proposed allowing States to use these funds for State-level activities or to make their own formula or competitive subgrants. Another commenter asked whether LEAs can serve non-Title I schools in their districts with their 50 percent share, and whether use of these funds must also adhere to Title I regulations. Discussion: Section 14006(c) of the ARRA requires at least 50 percent of Race to the Top funding to States to be sub-granted to participating LEAs according to their relative shares of funding under the ESEA Title I, Part A program for the most recent year. Neither the Department nor the States have discretion to deviate from this allocation requirement. LEAs that agree to work with the State to implement the State's Race to the Top plan may use these funds to serve non-Title I schools. Because these are not Title I program funds, LEAs are not required to adhere to Title I regulations regarding the usage of those funds. Fund uses, however, must be consistent with the State's plan and the Department's general regulations on uses of funds. In addition, States have considerable flexibility in awarding or allocating the remaining 50 percent of their Race to the Top awards, which are available for State-level activities, disbursements to LEAs, and other purposes as the State may propose in its plan. Many of the activities recommended by commenters would be allowable uses of the State's share of Race to the Top funds, including: Serving high-need students in non-Title I schools, State-level activities in support of Race to the Top plans, competitive or formula-based subgrants to LEAs, contracts with non-profit organizations, or supporting the participation of private school students and teachers in Race to the Top. Changes: None. Comment: One commenter stated that a portion of the Race to the Top funds should be set aside for LEA-IHE consortia to develop training that would allow for the development and implementation of systemic P- 20 collaboration, facilitate curricular alignment, and promote seamless transitions from high school to college. Discussion: As noted in the previous comment, section 14006(c) of the ARRA requires a State that receives a Race to the Top grant to use at least 50 percent of the award to provide subgrants to LEAs, including public charter schools identified as LEAs under State law. The ARRA does not require or specify that funds should be set aside for any other specific purposes; therefore, we decline to require that a portion of the Race to the Top funds be set aside for LEA-IHE consortia as recommended by the commenter. However, States are welcome to include such expenditures in their proposals if they align with their plans. We also note that IHEs are critical partners in implementing significant reforms, particularly in ensuring that a State's longitudinal data system can provide data to assess the extent to which students are adequately prepared for success in post-secondary education. As noted elsewhere, we are adding language to criterion (B)(3) to acknowledge the role that IHEs may play in supporting the transition to enhanced standards and high-quality assessments. In addition, as noted elsewhere, we are adding ``institutions of higher education'' in criterion (A)(2)(ii)(b) as an example of a type of stakeholder from whom a State should enlist support and commitment to assist in the State's education reform efforts. Changes: None. Sustaining Race to the Top Reforms Comment: One commenter expressed concern that the requirements and activities proposed in Race to the Top would not be fully paid for by Race to the Top awards, and that these activities would ``be difficult to sustain operationally and financially.'' This commenter recommended a sharper focus in the final notice on the requirements ``of greatest importance.'' In a related comment, one individual described Race to the Top as an ``underfunded mandate'' and argued that it would impose additional costs on State and local taxpayers. Discussion: While the Race to the Top program is intended to support a comprehensive approach to developing and carrying out critical change and reform in the four assurance areas, States have flexibility to tailor their Race to the Top budgets and spending plans according to both the relative priority of plan activities and the availability of funding from other Federal, State, and local sources, consistent with criterion (A)(2)(i)(d) (proposed criterion (E)(5)(v)). For example, States may use their Title I School Improvement Grants to execute most of their plans under criterion (E)(2) (proposed criterion (D)(3)), thereby allowing themselves to dedicate a higher proportion of Race to the Top funds to activities in the other three assurance areas. Similarly, a State that receives a Statewide Longitudinal Data Systems grant might use these funds to enhance its data systems work and could, therefore, focus its Race to the Top funding on other assurance areas. Also, the selection criteria include elements intended to help ensure that funding issues do not derail Race to the Top plans. For example, under criterion (F)(1), States are asked to demonstrate the extent to which (i) the share of overall State revenues supporting education in FY 2009 was greater than or equal to the share provided for education in FY 2008; and (ii) the State's policies lead to equitable funding (a) between high-need LEAs and other LEAs, and (b) within LEAs, between high-poverty schools and other schools (new criterion). In addition, criterion (A)(2)(i)(e) (proposed criterion (E)(5)(iii)) addresses whether a State has explained in its application how it will use its fiscal, political, and human capital resources to continue Race to the Top reforms after the period of funding has ended. Finally, because the Race to the Top is a voluntary, competitive grant program, it does not impose costs on any State or local taxpayers, and thus does not meet any reasonable definition of an underfunded mandate. Changes: Criteria related to budget planning and funding have been modified and rearranged in this final notice to promote the development and submission of more coherent Race to the Top plans. Criterion (A)(2)(i)(d) asks States to demonstrate through their budget narratives and accompanying budgets the extent to which they have high-quality plans to use Race to the Top funds to accomplish their plans and meet their targets, including, where feasible, coordinating, reallocating, or [[Page 59703]] repurposing education funds from other Federal, State, and local sources to align with their Race to the Top goals. Criterion (A)(2)(e) (proposed criterion (E)(5)(iii)) will help ensure that States have plans to continue support for Race to the Top reforms once Race to the Top funds have been spent. Addressing Obstacles Created by Poverty Comment: One commenter asserted that overcoming achievement gaps--a key goal of the Race to the Top program--would require addressing obstacles to high academic achievement created by the conditions of poverty. This commenter urged that Race to the Top be used to promote ``comprehensive educational opportunity'' for all students, but particularly for those from low-income families. Other commenters argued that Race to the Top plans should include efforts and incentives to ensure the adequacy and equity of State and local education funding, such as by rewarding States that have taken steps to allocate resources and inputs equitably. Discussion: The Secretary believes that a high-quality education is the surest route out of poverty. However, while broader societal problems such as the lack of affordable housing or access to health care certainly make the jobs of schools serving disadvantaged students more challenging, they should not be used to excuse the lack of achievement in high-need schools. Race to the Top is structured to promote comprehensive educational reforms benefitting all students while targeting additional attention and resources towards high-need LEAs and toward the persistently lowest-achieving schools that typically enroll a disproportionate number of students from low-income families. For example, 50 percent of Race to the Top funding must be subgranted by States to LEAs on the basis of their relative shares of formula grant allocations under Title I, Part A of the ESEA, which are based largely on counts of children from low-income families residing in the communities served by those LEAs. Also, under criterion (E)(2) (proposed criterion (D)(3)), States will create comprehensive school intervention plans for the persistently lowest-achieving schools. Furthermore, under criterion (D)(3) (proposed criterion (C)(3)), States will be evaluated on their plans to ensure that students in high- poverty and/or high-minority schools have equitable access to highly effective teachers and principals and are not served by ineffective teachers and principals at higher rates than other students. However, we agree that in this final notice, the Department should place greater emphasis on equitable funding of high-need LEAs and students. For this reason, we are adding criterion (F)(1)(ii), which examines the extent to which a State's policies lead to equitable funding (a) between high-need LEAs (as defined in this notice) and other LEAs, and (b) within LEAs, between high-poverty schools (as defined in this notice) and other schools. Changes: The addition of criterion (F)(1)(ii) establishes a new State Reform Condition Criterion that will consider the extent to which a State's policies lead to equitable funding (a) between high-need LEAs and other LEAs, and (b) within LEAs, between high-poverty schools and other schools. Civil Rights Enforcement Comment: Several commenters raised concerns about the NPP as it relates to civil rights laws and discrimination based on race and sex in schools. One commenter recommended that the Department include language in the final notice reminding States of their obligations under anti-discrimination statutes, including Title IX of the Education Amendments Act of 1972. Discussion: The Department believes in promoting educational excellence throughout the Nation through vigorous enforcement of civil rights laws. The Department's Office for Civil Rights is specifically tasked with enforcing several Federal civil rights laws that prohibit discrimination in programs or activities that receive Federal financial assistance from the Department, and issuing guidance to school districts on how to comply with those laws. Since SEAs and LEAs are ongoing recipients of Federal financial assistance, they are aware of these civil rights laws. We believe, therefore, that reiteration of State responsibilities under various civil rights laws in the final notice is unnecessary. Changes: None. Comment: One commenter suggested that the notice include language requiring States to support voluntary school integration efforts. Another commenter recommended adding an invitational priority for innovative approaches to voluntary school integration in order to encourage inter-district magnet schools and new charter schools that achieve racial and economic integration. The commenter also recommended adding an invitational priority to encourage the use of inter-district school transfers to promote integration. Another commenter recommended adding a criterion requiring a high-quality plan for a State to substantially reduce the isolation and segregation of low-income students, through intra- or inter-district collaboration, magnet schools, transfer programs, or school restructuring and consolidation. One commenter suggested adding requirements that State proposals reduce school-based poverty concentrations and racial isolation in schools. Another commenter wrote that the NPP overlooked ``the continuing importance of avoiding racial and economic segregation in public schools, and promoting voluntary integration'' and urged that the final notice promote these goals. Discussion: Racial and economic diversity are laudable goals that the Department supports. The Race to the Top program encourages innovative solutions to important problems facing our Nation's schools, which could include appropriate approaches to further racially and economically diverse schools. However, we have not added this objective as an invitational priority in the Race to the Top program. We note that the Department has for many years administered the statutory Magnet Schools Assistance Program, 20 U.S.C. 7231. This program provides grants to LEAs to fund magnet schools that--in addition to strengthening students' academic knowledge and their attainment of tangible and marketable skills--will further the ``elimination, reduction or prevention of minority group isolation'' in elementary and secondary schools. 20 U.S.C. 7231(b). Changes: None. Family and Community Engagement Comment: Many commenters stressed the importance of including parents, students, family, and community members ``as equal partners'' in developing States' Race to the Top plans. One commenter urged that the final notice require States and LEAs to document the involvement of parents in developing their Race to the Top plans, while another commenter recommended the inclusion of parent and student accountability measures in Race to the Top plans. One commenter urged that the Department and participating States keep parents informed of Race to the Top activities using materials written in ``easy-to- understand language'' and, where necessary, multiple languages. Several commenters stated that family engagement policies and practices that are culturally and linguistically appropriate are essential components of comprehensive services to high-need [[Page 59704]] students. A few commenters recommended that school personnel work with community partners to align school, family, and community assets and expertise in order to support student achievement (e.g., centers of community, community schools, community learning centers, full service community schools). Many commenters stressed the importance of family and community involvement in local school turnaround strategies. Several commenters also noted that the terms ``family engagement'' and ``community engagement'' should be separated, arguing that these concepts involve different stakeholders and require different strategies. Discussion: The Department agrees that States' Race to the Top plans would benefit from documented input and involvement by parents and organizations that represent parents, students, families, and community members. To encourage States to do so, we are adding, in criterion (A)(2)(ii)(b) (proposed criterion (E)(3)(ii)), Tribal schools; and parent, student, and community organizations among the stakeholders from which a State could obtain statements or actions of support to demonstrate statewide commitment to its Race to the Top plan. At the local level, criterion (E)(2) and Appendix C (proposed criterion (D)(3)) support greater parent involvement in individual school turnaround plans and the turnaround model and the transformation model in particular. The Department views such mechanisms not only as opportunities for parents to participate in turnaround planning but also for LEAs and schools to promote greater accountability for parents and students in areas such as school attendance, homework completion, and monitoring student achievement. In addition, the Department believes that any mechanism for family and community engagement naturally would require keeping parents informed of Race to the Top- related activities, including providing information in multiple formats and languages, where necessary. However, the final notice retains flexibility for LEAs to determine the nature of these mechanisms and does not specifically require plans to include separate parental involvement programs. Changes: Criterion (A)(2)(ii)(b) adds ``Tribal schools; parent, student, and community organizations (e.g., parent-teacher associations, nonprofit organizations, local education foundations, and community-based organizations)'' to the list of stakeholder groups from which a State can obtain statements or actions of support in order to demonstrate statewide support for its Race to the Top plan. I. Final Priorities General Comments on Proposed Priorities Comment: We received a number of comments that addressed more than one proposed priority or that focused on a proposed priority as well as on specific selection criteria. Discussion: In some cases we have responded to comments received in response to more than one priority or that focused on a priority and selection criteria in this ``General Comments on Proposed Priorities'' section. In other cases, we decided that it would be more appropriate to respond to the comments in the ``General Comments on the Race to the Top Program'' earlier in this notice. This enabled us to group similar comments and concerns in order to be more responsive to the commenters. Changes: None. Comment: One commenter stated that including absolute, competitive preference, and invitational priorities in the NPP was confusing and undermined the review process by suggesting that the Department does not have a clear sense of what is important. Another commenter recommended eliminating the invitational priorities claiming that they provide no competitive advantage in the grant competition and distract from the key elements of the program. One commenter requested that the final notice include an explanation of the differences and significance of the competitive preference priority for STEM and the invitational priorities for data systems, P-20 coordination, and school-level conditions for reform and innovation. Another commenter asked whether different weights will be assigned to the absolute priority versus the competitive preference and invitational priorities. Two commenters expressed concern with the statement in the NPP that the Secretary reserves the right to propose additional priorities, requirements, definitions, or selection criteria. These commenters requested that any additional priorities, requirements, definitions, or selection criteria be published in the Federal Register and that the public be given the opportunity to comment on them. Discussion: The Education Department General Administrative Regulations (EDGAR) in 34 CFR 75.105(c) identify the types of priorities the Department may establish for its direct grant programs. Under an absolute priority, the Secretary considers only those applications that meet the priority (see 34 CFR 75.105(c)(3)). Under a competitive preference priority, the Secretary may award bonus points to an application depending on the extent to which the application meets the priority or may select an application that meets the priority over an application of comparable merit that does not meet the priority (see 34 CFR 75.105(c)(2)). And, under an invitational priority, the Secretary may simply invite applications that meet the priority; an application that meets the invitational priority, however, receives no competitive or absolute preference over other applications (see 34 CFR 75.105(c)(1)). The designation of priorities as invitational in the NPP and in this final notice demonstrates the Department's interest in particular topics or issues and applicants' interest in and capacity to address those areas. Applicants are not required to address these invitational priorities in their applications. Because the Department is interested in State focus and capacity in the areas identified as invitational priorities, we decline to remove them in this final notice. In this final notice, we are designating priority 1, Comprehensive Approach to Education Reform, as an absolute priority that all applicants must meet. Priority 2, Emphasis on Science, Technology, Engineering, and Mathematics (STEM), has been designated as a competitive preference priority for which a State can receive additional points (see Appendix B for the scoring rubric). Finally, we are including the following invitational priorities: Priority 3, Innovations for Improving Early Learning Outcomes; priority 4, Expansion and Adaptation of Longitudinal Data Systems; priority 5, P-20 Coordination, Vertical and Horizontal Alignment; and priority 6, School-Level Conditions for Reform, Innovation, and Learning. Unless certain exceptions apply, the Department must conduct notice-and- comment rulemaking when establishing absolute and competitive preference priorities. See 34 CFR 75.105(b)(2). Notice-and-comment rulemaking is not required for the Department to establish invitational priorities. See 34 CFR 75.105(b)(2)(i). As noted by one commenter, we stated in the NPP that the proposed priorities could be changed in the final notice, and that the Department may propose additional priorities, requirements, definitions, or selection criteria, subject to applicable rulemaking requirements. As indicated elsewhere, we are adding a new invitational priority 3, Innovations for [[Page 59705]] Improving Early Learning Outcomes, based on comments received on the NPP. Since the priority is invitational only, we were able to include it in this final notice without additional public comment. Changes: None. Comment: Several commenters recommended that invitational priorities 4, 5, and 6 be changed to competitive preference priorities given the importance of each of the priorities and the need for States to have an integrated and coordinated reform strategy. One commenter recommended that additional points be given to a State that demonstrates how all the invitational priorities are integrated in its overall reform strategy. Discussion: We believe that priorities 4, 5, and 6 are appropriately designated as invitational priorities. Although the Secretary is interested in receiving applications addressing these priorities, each of the priorities extends or complements the core reform work that States must already address in their applications. For example, priority 4, Expansion and Adaptation of Statewide Longitudinal Data Systems, extends States' core work in developing statewide longitudinal data systems; priority 5, P-20 Coordination, Vertical and Horizontal Alignment, complements States' core reform efforts in the K- 12 education systems and extends them to the larger P-20 education systems; and priority 6, School-level Conditions for Reform, Innovation, and Learning, is a natural extension of the work States are doing to create, through law, regulation, or policy, other conditions favorable to education reform or innovation that improve student outcomes. For these reasons, we do not believe that extra points should be awarded to applications that address the invitational priorities. Changes: None. Comment: One commenter recommended adding an invitational priority to support alternative governance structures. The commenter stated that in addition to charter schools, mayoral control, gubernatorial control, and State control have been effective in reforming public education. Discussion: As noted elsewhere, we are adding criterion (F)(2)(v) to give credit to States that enable LEAs to operate innovative, autonomous public schools other than charter schools. Changes: None. Literacy Comment: Numerous commenters recommended that the final notice include a competitive preference priority focused on literacy development for young children; reading and writing skills for young students; and higher-order literacy skills for adolescent students (e.g., ability to analyze diverse texts and write using critical reasoning). Many commenters also proposed that priority be given to States that prepare more students (particularly low-income students, English language learners, and students with disabilities) for success in school and for graduation from high school ready for college and work, and with skills to meet the literacy demands of high-growth, high-wage jobs. Another commenter suggested that the final notice include access to high-quality school libraries as part of the criteria. Discussion: Advancing the literacy skills of all students, particularly students from low-income families, English language learners, and students with disabilities, is the foundation for many of the criteria in the Race to the Top competition. For example, a State will be judged on the extent to which it has made progress over the past several years in each of the four education reform areas, and used its ARRA and other Federal and State funding to pursue such reforms (see criterion (A)(3)(i)). A State will be judged on the extent to which it has demonstrated a track record of improving student achievement overall and by student subgroup in reading/language arts and mathematics, decreasing the achievement gaps between subgroups in reading/language arts and mathematics, and increasing high school graduation rates (see criterion (A)(3)(ii)). We believe that applicants must necessarily place priority on improving and advancing the literacy skills of students if they are to adequately address these criteria, and, therefore, do not believe that a separate competitive preference priority focused on literacy is necessary. Additionally, States and LEAs may determine in partnership the roles school libraries can play in advancing the State's reform goals. Changes: None. Early Learning Comment: Numerous commenters expressed concern that the NPP did not include a priority for, or otherwise require applicants to address, early learning in the context of the four reform areas. Several commenters highlighted the importance of early childhood education in improving student achievement and closing achievement gaps, and some cited research indicating that the most effective time to intervene to close achievement gaps is during the preschool years. Many commenters requested that the final notice include a competitive preference priority focused on early learning programs. One commenter stated that a competitive preference priority on early learning should focus on increasing the number of low-income children in high-quality pre-K programs. Other commenters recommended requiring a quality early learning strategy as part of a State's plan for turning around struggling schools. A number of commenters suggested that such a strategy could include expanded pre-K funding and programs, aligned standards and assessments for pre-K through third grade, links between longitudinal data systems and pilot ``Quality Rating and Improvement Systems'' to improve instruction, and increasing the availability of credentialed pre-K through third-grade teachers. Another commenter recommended that States be required to address the following issues to strengthen the quality of early care and education programs: (1) Appropriate compensation to attract and retain talented administrators and teachers in early care and education programs; (2) the need for a technological infrastructure to establish a data-driven decision-making system, as well as to document the benefits of early care and education services; (3) creation of a State- level advisory body to develop a State early learning plan, monitor the implementation of the plan and recommend adjustments to strengthen strategies as the plan is implemented; and (4) creation of a panel, that includes providers, to determine the true cost of supporting a quality early care and education system. A few commenters recommended adding an invitational priority to the final notice focusing on the coordination of preschool services (including Head Start services and services provided under the Individuals with Disabilities Education Act (IDEA)) in order to ensure that more young children begin school ready to learn. Discussion: The Department agrees that expanding access to high- quality early learning programs is a key strategy in an overall effort to raise student achievement, particularly for high-need students. We agree that the Race to the Top program should encourage States to increase the quality of existing early learning programs and expand access to high-quality early learning programs, particularly for children from low-income families. Therefore, we are adding an invitational priority focused on early learning to this final notice. [[Page 59706]] We do not believe that States should be required to include an early learning focus in their applications or that States should be given competitive preference points for doing so. Nor do we believe that quality early learning strategies should be required to be part of a State's plan for turning around struggling schools, given that efforts to turn around struggling schools focus primarily on improving educational outcomes for students currently enrolled in the Nation's persistently lowest-achieving schools. We believe that an invitational priority will encourage applicants to consider how their reform efforts can be strengthened by focusing on activities that promote school readiness and ensure that all children have access to high-quality early learning programs. With regard to the request that States be required to address the issues that one commenter stated were necessary for strengthening the quality of early care and education programs, a State that chooses to include a focus on early learning in its application could include activities addressing the educational needs of young children in its State reform plan. We note, however, that funds could not be used to address issues related to early child care needs, absent an educational component, because the purpose of Race to the Top is for States and LEAs to address educational reforms. Given the variation in State needs and priorities, we do not believe that it would be appropriate to require all applicants to follow the commenter's recommendations. In response to the recommendation to add an invitational priority focusing on the coordination of preschool services, this focus is already included in priority 5, P-20 Coordination, Vertical and Horizontal Alignment, which encourages State reform plans to address how early childhood programs, K-12 schools, postsecondary institutions, and other State agencies and community partners will coordinate to create a more seamless P-20 route for students. Changes: We have added a new invitational priority 3--Innovations for Improving Early Learning Outcomes, which states, ``The Secretary is particularly interested in applications that include practices, strategies, or programs to improve educational outcomes for high-need students who are young children (pre-kindergarten through third grade) by enhancing the quality of preschool programs. Of particular interest are proposals that support practices that (i) improve school readiness (including social, emotional, and cognitive); and (ii) improve the transition between preschool and kindergarten.'' School Climate and Culture Comment: Several commenters recommended that the final notice include a priority to encourage States to implement policies and take actions intended to improve school climate, such as citizenship training, anti-bullying, or service learning programs that may improve academic achievement, school attendance, and graduation rates. One commenter recommended adding an invitational priority for States that implement evidence-based measures to improve student discipline, stating that there is a well-documented link between school safety/ school discipline and improved academic outcomes. Several commenters specifically recommended that we provide for States to address school- wide systems of positive behavioral interventions and supports and stated that improving school climate is integral to improving the achievement of the lowest performing students. Another commenter stated that unless the Department designates school climate as a top priority, equal to that of academic improvement, schools are extremely unlikely to focus on improving school climate. A few commenters recommended encouraging States to collect data on school environments. Other commenters suggested that States support and recognize schools that provide opportunities for students to practice their education in real- world situations that lead to civic engagement. The commenters stated that States should ensure that, in policy and funding decisions, schools know that they are to be honored, as well as held accountable, for creating a caring, welcoming, safe environment. Other commenters strongly recommended that the final notice include language that would require schools to address the needs of the whole child, including by providing character education; instruction in social, emotional, and physical wellness; civic education and engagement; arts education; community-based learning; and opportunities for parent involvement. One commenter stated that it is essential for schools to work in collaboration with health, social, civic, faith- based, business and community organizations in order to successfully educate the whole child. One commenter expressed concern that the proposed priorities emphasize math, reading, and science at the expense of the other core academic subjects and argued that there should be an equal emphasis on the social, emotional, and creative development of students. Another commenter stated that efforts to shift education to address the needs of the whole child should be part of, and fully integrated into, a well-rounded core curriculum of academic instruction. Finally, one commenter stated that the proposed priorities incorrectly omit any reference to reducing the use of punitive measures in schools, and recommended that the final notice emphasize the Secretary's policy on reducing the use of restraints, seclusion, and corporal punishment. Discussion: We agree that a positive school climate that includes policies and measures to improve discipline can contribute to improving academic achievement, school attendance, and graduation rates. We also agree that it is important to address the needs of the whole child and to work in collaboration with other agencies and community organizations in order to successfully educate the whole child. Therefore, we are changing priority 6, School-Level Conditions for Reform, Innovation, and Learning to include school climate and school culture as examples of areas in which an LEA could provide flexibility and autonomy to its schools in order to create conditions for reform, innovation, and learning. The language in new paragraph (vi) of this priority acknowledges the importance of creating school climates and cultures that remove obstacles to, and actively support, student engagement and achievement; the language in new paragraph (vii) of the priority focuses on implementing strategies to effectively engage families and communities in supporting the academic success of their students. In addition, we note that the final notice addresses issues of school climate and culture in several ways. First, invitational priority 4, Expansion and Adaptation of Statewide Longitudinal Data Systems, invites States to include school climate and culture measures in extending and adapting their statewide longitudinal data systems. Consistent with commenters' examples of school policies and programs to improve school climate, we also have included references to ``service learning'' and ``experiential and work-based learning opportunities'' in the definition of increased learning time, as examples of activities that contribute to a well-rounded education. And we have included in our school intervention turnaround and transformation models for the persistently lowest-achieving schools (see criterion (E)(2) and [[Page 59707]] Appendix C) the need to address students' social and emotional needs and to create healthy school climates and cultures. We do not, therefore, believe that a new separate priority focusing on school climate and culture is necessary. We acknowledge that positive behavioral interventions and supports, as well as other systemic programs and policies that address bullying, student harassment, and disciplinary problems, are important to consider in ensuring that students have a safe and supportive environment in which to learn. However, we do not believe it is necessary to include this level of detail in this final notice and, therefore, decline to make the changes requested by the commenters. Finally, in response to the comment that the notice does not reference reducing the use of punitive measures, on July 31, 2009, the Secretary encouraged each State to review its current policies and guidelines regarding the use of restraints and seclusion in schools to ensure that every student is safe and protected and, if appropriate, develop or revise its policies and guidelines. We believe that this is the proper approach to addressing this issue, rather than in a notice for a competitive grant program for which all States will not necessarily apply or receive funding. It would be appropriate for States that choose to address priority 6 to include, in their reform plans, a focus on ensuring that policies and guidelines address the use of restraints and seclusions in schools to ensure that every student is safe and protected. Changes: We have revised priority 6 to include as examples of the autonomies and flexibilities a State's participating LEAs may provide to its schools: Creating school climates and cultures that remove obstacles to, and actively support, student engagement and achievement and implementing strategies to effectively engage families and communities in supporting the academic success of their students. Charter Schools Comment: Several commenters recommended that the final notice include an absolute priority requiring States to expand charter schools. Discussion: We do not believe an absolute priority for charter schools is necessary because States already will be evaluated against criteria that support the development of high-quality charter schools. Criterion (F)(2) focuses on charter schools. Specifically, criterion (F)(2)(i) considers the extent to which a State has a charter school law that does not prohibit or effectively inhibit increasing the number of high-performing charter schools in the State or otherwise restrict student enrollment in charter schools. Criterion (F)(2)(ii) considers the extent to which the State has laws, statutes, regulations, or guidelines regarding how charter school authorizers approve, monitor, hold accountable, reauthorize, and close charter schools. Under criterion (F)(2)(iii), a State will be evaluated based on the extent to which its charter schools receive equitable funding and a commensurate share of local, State, and Federal revenues. Finally, criterion (F)(2)(iv) addresses the extent to which a State provides charter schools with funding for facilities, assistance with facilities acquisition, access to public facilities, the ability to share in bonds and mill levies, or other supports; and the extent to which a State does not impose any facility-related requirements on charter schools that are stricter than those applied to traditional public schools. All applicants will be rated against these criteria, among others. Changes: None. Dropout Recovery Comment: One commenter expressed concern that the NPP did not include targeted investments for dropout recovery programs or provide States and LEAs with direction on innovative models to re-engage youth who have dropped out of school. The commenter stated that the recovery of high school dropouts must be a central component of any serious systemic school reform effort. Several commenters stated that it is important to recognize that students who fail to thrive in traditional settings need additional supports to graduate from high school and that, without strategic approaches that intentionally include re- engagement efforts, districts will not serve this population effectively. Another commenter recommended that the final notice include a competitive preference priority for serving students who are still in school, but are off-track to graduate and those who have disengaged from school and dropped out. The commenter noted that educational continuity and stability are also needed for children in foster care. One commenter recommended establishing a competitive preference priority for applicants that include data-driven strategies to re- engage high-school students who fail to graduate on time and recommended that the final notice encourage States to coordinate Race to the Top funding with funding they receive through other sources such as programs under the Workforce Investment Act. Discussion: We agree that there is a need to increase efforts to re-engage youth who have dropped out of school and to help students who are off-track to graduate stay in school. We have addressed the needs of these students in several ways. First, as noted elsewhere, we are changing criterion (E)(2) (regarding States' plans to enable their LEAs to implement one of the four school intervention models) to include credit-recovery programs and re-engagement strategies as methods that can be used by LEAs to increase high school graduation rates (see Appendix C). Second, we are adding a new definition of high-need students and including in the definition, among others, students who are performing far below grade level, those who leave school before receiving a regular high school diploma, and those at risk of not graduating with a diploma on time. Third, as noted in the discussion of priority 4, we are inviting States to extend and adapt their statewide longitudinal data systems to include data from programs that serve at- risk students and from dropout prevention programs. Fourth, we are adding a reference to horizontal alignment in priority 5. Horizontal alignment is the coordination of services across schools, State agencies, and community partners, and we note that it is important in ensuring that high-need students have access to the broad array of opportunities and services they need and that are beyond the capacity of a school itself to provide. We also note that priority 6, School- Level Conditions for Reform, Innovation, and Learning, specifically refers to the need to provide comprehensive services to high-need students (see paragraph (v)). Therefore, we believe that this final notice adequately addresses the needs of students off-track to graduate who are still in school and those who have disengaged from school and dropped out, and that it is unnecessary to add a competitive preference priority focused on these specific youth. With regard to the comment that the final notice encourage coordinating ARRA funding with other funding streams, we believe this issue is addressed in criterion (A)(2)(i)(d), which will evaluate the extent to which a State has the capacity to use Race to the Top funds, as described in the State's budget and budget narrative, to accomplish the State's plan and meet its targets, including, where feasible, by coordinating, reallocating, or ``repurposing'' education funds from other Federal, State, and local sources to [[Page 59708]] align with the State's Race to the Top goals. Changes: None. Students With Disabilities and English Language Learners Comment: One commenter encouraged the Department to add invitational priorities that focus on policy development and implementation (versus data collection and analysis) for special education and English language acquisition, including the development of high-quality and innovative programs of teacher preparation and professional development in these areas, in order to encourage States to meet the needs of students with disabilities and English language learners more effectively. Another commenter expressed disappointment that the priorities did not thoroughly take into account the needs of English language learners. One commenter strongly urged the Department to ensure that English language learners are not overlooked in State plans, but are explicitly identified in all areas, including through efforts to improve standards and assessments, close achievement gaps, increase graduation rates, and ensure college readiness. Discussion: The needs of students with disabilities and English language learners are addressed in many of the selection criteria and are especially highlighted everywhere the term high-need student is used; the new definition of this term includes students with disabilities and English language learners. All applicants for Race to the Top grants will need to consider how they currently work to meet or plan to meet the unique needs of these students based on the criteria set forth in this final notice. In addition, this final notice recognizes and specifically references the unique needs of students with disabilities and English language learners in the following areas: (a) Priority 4 encourages State plans to expand statewide longitudinal data systems to include or integrate data from special education and English language learner programs; (b) criterion (C)(3)(iii) will be used to assess the extent to which States make their data systems available and accessible to researchers so that they have information to evaluate the effectiveness of instructional materials, strategies, and approaches for educating different types of students, such as students with disabilities and English language learners; and (c) criterion (D)(3) will be used to examine States' plans to increase the number and percentage of highly effective teachers teaching in hard-to-staff subjects and specialty areas, such as special education and language instruction educational programs (as defined under Title III of the ESEA). In addition, the measures used to document increases in achievement, closing achievement gaps, and increasing graduation rates, all require data to be disaggregated by subgroups, including the students with disabilities and limited English proficient students subgroups (see criteria (A)(1)(iii) and (A)(3)(ii)). Therefore, we believe that this final notice ensures that students with disabilities and English language learners are not overlooked in State reform plans and that it is unnecessary to add an invitational priority focused on students with disabilities and English language learners. Changes: None. Curriculum, Instruction, Assessments, Professional Development Comment: One commenter stated that the proposed priorities have little to do with improving curriculum, instruction, assessments, or professional development and recommended that in the final notice, the Department give priority to developing and implementing core school improvement activities, particularly school-based collaborative activities to improve teaching. Discussion: We disagree with the commenter's statement that the proposed priorities have little to do with improving curriculum, instruction, assessments, or professional development. In order to receive a Race to the Top grant, States must demonstrate that they have made and will continue to drive significant improvement in student outcomes, including making substantial gains in student achievement, closing achievement gaps, improving high school graduation rates, and ensuring that students are prepared for success in college and careers. To accomplish this, a State would have to focus on improving curriculum, instruction, assessments, and professional development. Furthermore, absolute priority 1 requires all applicants to address comprehensively each of the four education reform areas specified in the ARRA--enhancing standards and assessments, improving the collection and use of data, increasing teacher effectiveness and achieving equity in teacher distribution, and turning around struggling schools. In addressing each of these reform areas, States will necessarily have to focus on improving curriculum, instruction, assessments, and professional development. Furthermore, criteria (B)(3), (C)(3)(ii), (D)(2)(iv)(a), and (D)(5) explicitly focus on professional development. Criterion (B)(3) focuses on, among other activities, professional development to support the transition to new standards and assessments; as noted elsewhere, criterion (C)(3)(ii) has been added to focus on professional development for teachers, principals and administrators on using instructional improvement systems to support continuous instructional improvement; criterion (D)(2)(iv)(a) refers to using teacher and principal evaluations to inform relevant professional development; and criterion (D)(5) focuses on the need for States and LEAs to provide effective data-informed professional development, coaching, induction, and common planning and collaboration time to teachers and principals that are, where appropriate, ongoing and job-embedded. Changes: None. Research-Based Practice Comment: One commenter recommended adding an invitational priority to encourage States to adopt programs that have been demonstrated to be effective through rigorous research. The commenter stated that priority should be given to States that identify resources to help their LEAs select programs that are supported by the best available empirical evidence. Discussion: Criterion (A)(2)(i)(b) will be used to judge the extent to which a State has the capacity to support its participating LEAs in successfully implementing the education reform plans the State has proposed through such activities as identifying promising practices, evaluating these practices' effectiveness, and ceasing ineffective practices. In addition, criteria (C)(2) and (C)(3) focus on gathering and using data to support continuous improvement, including a specific focus on making the data available and accessible to researchers to evaluate the effectiveness of instructional materials, strategies, and approaches. We believe these criteria address the commenter's concerns and, therefore, that it is unnecessary to add the invitational priority suggested by the commenter. Changes: None. Using Data To Inform Practice Comment: One commenter urged the Department to add a competitive preference priority for establishing an ``evidence-based learning cycle'' to improve system-wide policy and student achievement results. The commenter recommended that the [[Page 59709]] competitive preference priority encourage States to: (1) Design robust formative and summative evaluations on their Race to the Top programs; (2) gather data on the highest-priority teacher and principal actions, and school-level and classroom-level practices that differentiate fast- improving schools and classrooms from other schools and classrooms; and (3) document these practices so that other teachers, school leaders, and State and local policymakers can access and use these tools and evidence to drive a continuous cycle of improvement in other schools, classrooms, and systems. Another commenter recommended adding the development of longitudinal data systems as a competitive preference priority in order to accelerate development and implementation of next-generation, user- oriented data systems that provide timely, useful data for teachers and principals to use in managing performance and improving student achievement; prioritize academic data with an emphasis on leading predictive indicators; include routine data inquiry processes and training to support educators in the effective interpretation and use of data that result in improved student achievement; and enhance State and local capacity to use data and improve the systematic integration and use of data over time. Discussion: The evidence-based learning cycle and the user-oriented data systems proposed by the commenters are similar in concept to criteria (C)(2) and (C)(3). Criteria (C)(2) and (C)(3) focus on the use of data from the State's statewide longitudinal data system and the local instructional improvement systems to support continuous improvement both within and outside of the classroom. In addition, priority 4 focuses on expanding statewide longitudinal data systems to include or integrate data from a variety of sources, including, for example, human resources, school finance, and other relevant areas with the purpose of connecting and coordinating all parts of the system to inform continuous improvement practices. Therefore, we do not believe it is necessary to make the changes recommended by the commenters. Changes: None. Flexibility in Operating Conditions Comment: One commenter recommended that the Department include an invitational priority for applicants that commit to implementing the reforms and providing flexible operating conditions for their schools. Discussion: We agree that flexibility in operating conditions is an important strategy to facilitate reform efforts. That is why we included priority 6, School-Level Conditions for Reform, Innovation, and Learning, which focuses on flexibilities and autonomies that an LEA provides to its schools in order to create the conditions for reform, innovation, and learning. Changes: None. Priority 1: Absolute Priority--Comprehensive Approach to Education Reform: General Comments Comment: Numerous commenters expressed support for absolute priority 1 and its focus on ensuring that States comprehensively address each of the four education reform areas and take a systemic approach to education reform. The commenters stated that this approach will encourage school systems around the country to implement much- needed changes that will improve student outcomes. One commenter stated that this approach sets a much higher bar for State applications than is typically required of competitive grant programs and was supportive of this approach. Another commenter encouraged the Department to award Race to the Top grants only to those States that pursue significant comprehensive and systemic reforms. However, one commenter expressed concern that this approach would encourage States to lower standards rather than provide incentives for States to improve their educational standards and put in place the reforms necessary to improve educational outcomes. Discussion: We appreciate the support for absolute priority 1 and its focus on a comprehensive and systematic approach to addressing the four education reform areas specified in the ARRA. We do not agree with the commenter that a comprehensive and systematic approach to the four reform areas will encourage States to lower standards. The focus on improving student achievement, decreasing achievement gaps, and increasing high school graduation rates, and the use of sound measures, such as the results from the NAEP, will help ensure that States do not lower their standards. In addition, unlike in other competitive programs, we are rewarding States that have already created the conditions for reform and improved student outcomes and have a strong foundation for implementing plans going forward. States that have lowered their standards will not clear the high bar that we have set for awards under the Race to the Top program. As noted elsewhere, we are adding to this final notice a new section (A), State Success Factors. We are revising a number of the selection criteria from proposed section (E) (Overall Selection Criteria) and including them as State Success Factors Criteria (A). The purpose of this change is to provide States with the opportunity to begin their proposals with clear statements of their integrated, coordinated, statewide reform agendas. In order to be consistent with this change, we are changing the language in priority 1 to provide that, in addition to addressing the four education reform areas, State applications also must address the State Success Factors Criteria. Consistent with this focus on the State Success Factors Criteria, we are adding clarifying language and removing the reference to the four reform areas in the title of absolute priority 1. With regard to the use of NAEP scores to measure increasing student achievement, we are removing this reference in priority 1 because, as noted elsewhere, the new section on State Success Factors describes how increases in student achievement and closing achievement gaps across subgroups will be measured. State Success Factors Criteria (A)(1)(iii) and (A)(3)(ii) specify that when evaluating increases in student achievement and gap-closing, reviewers will examine results in reading/ language arts and mathematics based on the NAEP and on the assessments required under the ESEA. Changes: Absolute priority 1 has been revised to read: ``To meet this priority, the State's application must comprehensively and coherently address all of the four education reform areas specified in the ARRA as well as the State Success Factors Criteria in order to demonstrate that the State and its participating LEAs are taking a systemic approach to education reform. The State must demonstrate in its application sufficient LEA participation and commitment to successfully implement and achieve the goals in its plans; and it must describe how the State, in collaboration with its participating LEAs, will use Race to the Top and other funds to increase student achievement, decrease the achievement gaps across student subgroups, and increase the rates at which students graduate from high school prepared for college and careers.'' Competitive Preference Priority 2: Emphasis on Science, Technology, Engineering, and Mathematics (STEM): Comment: Numerous commenters expressed support for including an emphasis on STEM education as a competitive preference priority. The [[Page 59710]] commenters noted that major developments in medicine, energy, and agriculture are dependent on innovations in STEM fields and stated that engaging students in STEM education programs is the most effective way to improve the Nation's economy and maintain America's global leadership. One commenter recommended changing the priority to an absolute priority and another commenter recommended adding selection criteria related to STEM education. However, many commenters stated that designating STEM as a competitive preference priority implies that STEM subjects are more important than other subjects and recommended omitting or changing the STEM priority to an invitational priority. One commenter asked why the Department chose to emphasize STEM subjects over other subjects. Numerous commenters expressed concern that including a competitive preference priority on STEM education would lead to a narrowing of the curriculum. One commenter expressed concern that a competitive preference priority emphasizing STEM education might encourage STEM- only programs, as opposed to STEM-focused programs in which the content is integrated into various curricular areas. The commenter expressed concern that the priority would prohibit States from applying data- driven reform and school achievement interventions that do not focus on STEM. Another commenter recommended changing the priority to give States the option of using data to develop plans that meet the needs of their low-performing schools. Discussion: We appreciate the support expressed for including a competitive preference priority on STEM education. Ensuring American competitiveness in a global economy requires significant improvements in STEM education. As the commenters noted, professionals in STEM fields are major contributors to the American economy in such areas as medicine, agriculture, and energy. Science-based industries are in need of skilled workers, and we believe a competitive preference priority on STEM will help schools produce a generation of Americans who can meet this demand. Therefore, we decline to eliminate priority 2 or to re- designate priority 2 as an invitational priority. We did not intend for an emphasis on STEM education to result in a narrowing of the curriculum. Rather, our intent was to focus attention on the need to develop and implement rigorous courses of study in STEM fields, assist teachers in providing effective and relevant instruction in those fields, and prepare more students for advanced study and careers in STEM. While we believe increasing the focus on STEM education is important, we do not believe that an emphasis on STEM education should be required as part of the core work that States are required to address in their reform plans for the Race to the Top program. Therefore, we decline to change the emphasis on STEM education to an absolute priority or include selection criteria emphasizing STEM education. With regard to commenters' concerns that emphasizing STEM education might encourage STEM-only programs, as opposed to STEM- focused programs, we note that this notice specifically refers to preparing and assisting teachers in integrating STEM content across grades and disciplines. The priority will not prohibit States from using data from areas other than STEM education to drive reform, nor should it discourage them from doing so. Changes: None. Comment: Two commenters recommended that the final notice clarify the meaning of ``a rigorous course of study,'' as used in priority 2, by providing examples of what the Department considers to be rigorous courses of study. The commenters suggested Advanced Placement courses and STEM-intensive courses, such as those offered in many career and technical education programs, as examples of rigorous courses of study. One commenter recommended including a reference to career preparatory coursework. Two commenters recommended the final notice include an incentive for States that assess the alignment of rigorous courses of study in STEM subjects with other courses of study in a school's curriculum. Discussion: The Department believes that States should have the flexibility to determine the content and focus of a rigorous course of study in STEM subjects and, therefore, declines to add examples of rigorous courses of study in priority 2. In determining the rigor of a course in STEM subjects, local decision-makers will likely assess how STEM subjects are integrated and aligned with other courses offered in a State or LEAs' current programs of study. Therefore, we do not believe that it is necessary to provide incentives for doing so. Changes: None. Comment: A few commenters recommended that the final priority reference additional STEM-capable community partners such as youth- serving community organizations, ``valued-added intermediaries,'' and public broadcasting entities. One commenter strongly recommended that the Department provide guidelines for selecting STEM-capable partners. Another commenter noted that non-school settings, such as museums and science centers, offer designed spaces and programs to engage students and encourage them to pursue and develop interests in scientific inquiry that may positively influence academic achievement and expand students' sense of career options. Discussion: To meet priority 2, applicants must cooperate with industry experts, museums, universities, research centers, or other STEM-capable community partners in preparing and assisting teachers to integrate STEM content across grades and disciplines, to promote effective and relevant instruction, and to offer applied learning opportunities for students. We do not believe it is appropriate to be more specific about the STEM-capable partnerships that States should form given that the resources and needs vary considerably across schools and communities; such decisions are best left to local decision makers. Therefore, we decline to include additional examples of STEM- capable partnerships or to provide guidelines for selecting STEM- capable partners, as requested by commenters. Changes: None. Comment: One commenter recommended that the Department revise priority 2 to explicitly include computer science as part of STEM education. The commenter stated that computer science is often confused with technology literacy and this confusion leads to teaching basic skills instead of core concepts and problem solving. The commenter noted that computer science provides students with a fundamental understanding of computing, exposure to professional fields, and opportunities to develop computational thinking skills. Discussion: STEM education includes a wide-range of disciplines, including computer science. We believe that States should have the flexibility to define the specific courses of study in mathematics, the sciences, technology, and engineering, based on the needs and available resources of the State, as well as the advice of industry experts, museums, universities, research centers, and other STEM-capable community partners. Therefore, we decline to change priority 2 to specify that computer science is a part of STEM education, as requested by the commenter. Changes: None. [[Page 59711]] Comment: One commenter recommended that the Department require States to implement the recommendations of the National Mathematics Advisory Panel regarding K-8 mathematics teacher preparation programs and licensing requirements. The commenter stated that teacher preparation programs and licensing requirements for K-8 mathematics teachers should address arithmetic, geometry, measurement, and algebra. Another commenter recommended requiring States to provide funds for improving State licensing requirements in order to ensure that K-8 teachers master core mathematics content. One commenter recommended that the Department require in-service training for K-8 mathematics teachers. Another commenter recommended that the Department revise priority 2 in order to ensure that teachers in high-risk, low- performing schools are provided with professional development opportunities, mentoring, and the necessary guidance to ensure that rigorous courses of study in STEM subjects are taught in these schools. Discussion: We do not believe that it would be appropriate for the Department to require States to implement the recommendations of the National Mathematics Advisory Panel regarding mathematics teacher preparation programs and licensing requirements; decisions regarding teacher preparation programs and licensing requirements are best left to State and local officials to make depending on the unique needs and circumstances in each State. With regard to the recommendation to require in-service training and professional development, mentoring, and guidance in STEM subjects to teachers in high-risk, low-achieving schools, we note that this final notice includes several criteria that address the professional development needs of teachers, including criteria (B)(3), (C)(3)(ii), (D)(2)(iv)(a), and especially (D)(5), which focuses on the extent to which States provide effective support to teachers and principals. We believe that these criteria adequately address the commenter's concerns regarding professional development; States addressing the STEM competitive preference priority will have ample opportunities to address professional development needs in their responses to these criteria. We therefore decline to change priority 2 in the manner recommended by the commenter. Changes: None. Comment: A few commenters recommended that the Department encourage States to recruit, train, and provide alternative pathways for STEM professionals to join the teaching force as full-time teachers, co- teachers, or professional development providers. The commenters noted that STEM professionals in the classroom would help students understand the career opportunities available for individuals with knowledge in STEM subjects. One commenter recommended providing additional credit to States that use ``informal science education centers'' as resources for professional development. Discussion: We agree with commenters that efforts should be made to recruit and train STEM professionals to join the teaching force as teachers and that having such professionals in the classroom would help students understand the career opportunities available in STEM fields. Criterion (D)(1), which assesses the extent to which a State has high- quality pathways for aspiring teachers and principals, addresses this concern. To the extent that the informal science education centers, referred to by one commenter, provide professional development as an alternative route to certification, States that permit use of such centers would be given credit under criterion (D)(1)(i). Therefore, we decline to give additional credit to States that use such centers as recommended by one commenter. Changes: None. Comment: One commenter recommended that the Department invite States to strengthen their early childhood education programs by including STEM education in their State reform plans for early learning programs. Discussion: As noted elsewhere, we are adding an invitational priority for early learning programs (see priority 3), which includes a focus on improving young children's school readiness, and a competitive preference priority for STEM education (see priority 2). States that choose to address either of these priorities could include a description of efforts to ensure that early learning program standards and curricula include developmentally appropriate science, pre- numeracy, and numeracy content in order to help prepare young children to succeed in STEM-related areas when they enter school. Changes: None. Comment: One commenter recommended that the Department encourage States to provide high-level STEM curricula to advanced students in earlier grades than is typically the norm. The commenter noted that local policies and practices typically inhibit acceleration options and leave advanced students unchallenged. Discussion: With regard to the commenter's recommendation that the Department encourage States to provide high-level STEM curricula to advanced students in earlier grades than is typical, States will have opportunities to include such concepts in their applications, if they so desire, through priority 6, which focuses on LEAs creating the conditions for reform and innovation by providing their schools with flexibilities and autonomies; through criterion (B)(3), which addresses instructional issues relating to enhanced standards; and by addressing competitive preference priority 2, which focuses on STEM education. Changes: None. Comment: One commenter urged the Secretary to encourage States to open statewide, public, residential high schools that focus on math and science. Discussion: To the extent that a public residential high school would be considered an innovative school, we note that criterion (F)(2)(v) encourages States to enable LEAs to operate such innovative, autonomous public schools. Therefore, we do not believe that additional language in priority 2 is needed to address the commenter's recommendation. Changes: None. Comment: One commenter stated that the availability of up-to-date laboratory equipment plays an important role in STEM learning and requested that the Department clarify whether Race to the Top funds could be used to purchase laboratory equipment and technological tools to implement STEM programs. The commenter stated that the quality and quantity of equipment is inadequate in most schools, particularly in schools with high concentrations of at-risk students. Discussion: The Race to the Top program provides States and LEAs with significant freedom to use Race to the Top funds to meet the goals outlined in their State reform plans. Laboratory equipment would be an allowable use of funds under the Race to the Top program. Changes: None. Comment: One commenter urged the Department to encourage States to develop a common set of core STEM standards and assessments. In addition, the commenter recommended that the Department encourage and reward States that enhance their high school graduation requirements to include four years of STEM courses. Discussion: The Department is encouraging States to develop a common set of high-quality K-12 standards that are internationally benchmarked and that build toward [[Page 59712]] college- and career-readiness by the time of high school graduation. In addition, the Department is encouraging States to develop and implement common, high-quality assessments that are aligned with those standards. Thus, criterion (B)(1) assesses the extent to which a State has demonstrated its commitment to adopting a common set of high-quality standards, and criterion (B)(2) assesses the extent to which the State has demonstrated its commitment to improving the quality of its assessments. It is a State's responsibility to determine the content of those standards and assessments, including whether to develop a common set of core STEM standards and assessments. Likewise, States are responsible for establishing high school graduation requirements. Thus, whether or not four years of STEM courses are included as a requirement for graduation from high school is a decision that is made by States, not the Federal Government. Changes: None. Comment: Several commenters recommended that the Department require STEM instruction to be consistent with the principles of universal design for learning. The commenters noted that universal design for learning is defined in section 103(24) of the Higher Education Opportunity Act of 2008 (Pub. L. 110-315), as a structure that provides flexibility in instruction that accommodates, supports, and maintains high achievement expectations for all students, including students with disabilities and English language learners. Discussion: Paragraph (ii) in priority 2 focuses on promoting STEM education that is effective, relevant, and includes applied learning opportunities for students. To the extent that such instruction can be provided consistent with the principles of universal design, we encourage States to do so. However, we do not believe it would be appropriate to require all instruction to be consistent with the principles of universal design for learning as recommended by the commenters. Changes: None. Comment: A few commenters recommended that the Department promote racial, economic, and gender integration in STEM programs. These commenters stated that programs funded by the Department have an obligation to be inclusive and remove discriminatory barriers. One commenter noted that STEM programs should be included in schools that serve low-income students to ensure that such students have access to STEM programs. Another commenter recommended that the Department reiterate that recipients of Race to the Top funds should remove obstacles that might discourage female students from enrolling and completing STEM programs. Discussion: We agree with these commenters that all students should have access to rigorous courses of study in STEM programs. Paragraph (iii) in priority 2 specifically refers to State plans addressing the needs of underrepresented groups and of women and girls in the areas of science, technology, engineering, and mathematics. Therefore, we do not believe that additional language needs to be added to priority 2 to address the commenters' concerns. Changes: None. Comment: Two commenters recommended that the final notice reference advanced laboratory work, service learning, project-based learning, and work-based learning as examples of ``applied learning opportunities.'' The commenters stated that providing such examples would help clarify the meaning of applied learning opportunities as it is used in priority 2. One commenter recommended that the Department clarify that applied learning opportunities could occur during regular school hours, or before or after the regular school day. Discussion: A State seeking to meet priority 2 is required to cooperate with industry experts, museums, universities, research centers, and other STEM-capable community partners to ensure that instruction is relevant and that students are provided with opportunities to apply what they have learned in the classroom. Such cooperative work with experts in STEM fields should provide a State with ample examples of applied learning opportunities. In addition, as noted elsewhere, we are adding a definition of increased learning time; this definition specifically references service learning and experiential and work-based learning and encourages such learning to occur during or outside of regular school hours. As such, we do not believe it is necessary to include examples of applied learning opportunities in priority 2, which could limit, rather than promote ideas and strategies to improve or enhance STEM education programs. Changes: None. Comment: One commenter recommended that priority 2 be changed to require State reform plans to describe how technology will be incorporated as a required component in STEM education programs. The commenter also recommended requiring State reform plans to include online access to high-quality STEM courses and instructors, remediation for low-performing students through interactive instructional software, virtual field trips, and online connections to STEM professionals. Another commenter noted that programs supported by universities use technology and multimedia to improve teaching and learning of STEM subjects and recommended that universities and the business sector work in partnership with schools to prepare students for postsecondary education and workplace success. Discussion: We agree that the approaches that commenters discussed can be useful in implementing STEM programs. However, we believe such decisions are best left to local officials who understand the needs and available resources in their schools and communities. We decline, therefore, to make the changes that the commenters recommend. Changes: None. Comment: One commenter asked how the Department will determine whether a State's application meets the competitive preference priority. The commenter asked specifically whether a ``pilot'' project focused on STEM education, rather than a comprehensive STEM program, would meet priority 2. Another commenter recommended that the Department require a State's proposed STEM programs to be evidence- based. Discussion: Priority 2 describes the three elements that a State's reform plan must address to meet priority 2. These elements include the need to (i) offer a rigorous course of study in STEM subjects; (ii) cooperate with industry experts, museums, universities, research centers, or other STEM-capable community partners to prepare and assist teachers in integrating STEM content across grades and disciplines, in promoting effective and relevant instruction, and in offering applied learning opportunities; and (iii) prepare more students for advanced study and careers in science, technology, engineering, and mathematics, including by addressing the needs of underrepresented groups and of women and girls in STEM areas. We are clarifying that, to meet the priority, the State's application must have a high-quality plan to address each of these elements. We do not believe it is necessary to require that a State's proposed STEM program be evidence-based in order to meet this priority; reviewers will judge the quality of the program that a State proposes, which will necessarily include the extent to [[Page 59713]] which the State's proposed STEM education program is evidence-based. Changes: We have revised the priority to specify that, to meet this priority, the State's application must have a high-quality plan to address the areas specified in the priority. Comment: One commenter stated that a significant investment is necessary to successfully improve student performance in STEM subjects and recommended that the Department revise priority 2 to provide a preference to States with the infrastructure to demonstrate results. Discussion: We do not believe that preference should be given to States that already have the infrastructure in place to evaluate and demonstrate results. As part of its application, each State must provide a detailed budget and accompanying budget narrative describing how the State plans to use Race to the Top funds to accomplish the State's reform plan and meet its targets. The detailed plan for using grant funds must include, among other things, the key goals, the key activities to be undertaken, the rationale for the activities, and the timeline for implementing the activities (see application requirements). A State that includes a focus on STEM education must, therefore, include in its proposed budget how it plans to use grant funds or other Federal, State, and local funds to meet its goals related to improving STEM education. Changes: None. Priority 4--Invitational Priority--Expansion and Adaptation of Statewide Longitudinal Data Systems (Proposed Priority 3): Comment: A number of comments were received on priority 4 that were similar to the comments received on criterion (C)(1), regarding implementing a statewide longitudinal data system; criterion (C)(2), regarding accessing and using State data; and criterion (C)(3), regarding using data to improve instruction. Discussion: In some cases we have responded to comments received in response to priority 4 under section (C), Data Systems to Support Instruction. This enabled us to group similar comments and concerns in order to be more responsive to the commenters. Changes: None. Comment: One commenter recommended changing the title of this priority to ``Expansion, Adaptation, and Appropriate Utilization of State Longitudinal Data Systems.'' Discussion: We do not believe the lengthier title recommended by the commenter is necessary, and therefore, decline to change the title of priority 4. Changes: None. Comment: One commenter recommended that priority 4 be eliminated. The commenter stated that Race to the Top funds should be used to improve teaching and not for expanding data systems. Discussion: Establishing a statewide longitudinal data system that provides data on student achievement or student growth to teachers and principals, as well as policymakers, researchers, and other stakeholders, is key to driving education reform in general, and improvements in the classroom, in particular. Therefore, we decline to eliminate priority 4. Changes: None. Comment: Several commenters recommended that priority 4 be changed from an invitational priority to a competitive preference priority because of the importance of linking data from various program areas with statewide longitudinal data systems. Several commenters stated that expanding and linking data systems are essential to achieving comprehensive reform in the four ARRA education reform areas, and therefore, recommended changing the priority to an absolute priority. Discussion: We believe that priority 4 is appropriately designated as an invitational priority because it extends the work that States are already doing to address the criteria related to fully implementing statewide longitudinal data systems. A State will already be judged on the extent to which it has a statewide longitudinal data system that includes all of the America COMPETES Act elements (see criterion (C)(1)) and the extent to which it has a high-quality plan to ensure that data from the State's statewide longitudinal data system are used to support decision-makers in the continuous improvement of policy, instruction, operations, management, resource allocation, and overall effectiveness (see criterion (C)(2)). While we believe that the focus of priority 4 is important, it is not part of the core work that States must do to address the four education reform areas. Therefore, we decline to re-designate priority 4 as an absolute priority or as a competitive preference priority. Changes: None. Comment: One commenter requested clarification about the data that are required to meet this priority and the questions these data should be able to answer. Discussion: Criterion (C)(1) will examine the extent to which a State has a statewide longitudinal data system that includes all of the America COMPETES Act. The purpose of priority 4 is to reward States that go beyond the 12 elements of the America COMPETES Act to connect their statewide longitudinal data systems to other data or data systems that may exist independently from a State's statewide longitudinal data system. The information that will be responsive to this priority will depend on each State's current statewide longitudinal data system, the extent to which it is already connected to other data or data systems, and the types of questions related to policy, practice, or overall effectiveness that a State needs to answer in order to implement its reform agenda. We believe that this purpose could have been stated more clearly in the priority and, therefore, are adding clarifying language. Changes: We have changed the end of the last sentence in the first paragraph of the priority as follows: ``* * * with the purpose of connecting and coordinating all parts of the system to allow important questions related to policy, practice, or overall effectiveness to be asked, answered, and incorporated into effective continuous improvement practices.'' Comment: One commenter noted that statewide longitudinal data systems could be expanded in a number of ways such as including additional data from within the agency, from other State agencies, from other States, or from management systems that track and allocate resources. The commenter recommended that the priority include this clarification. Another commenter recommended that the priority encourage States to link their longitudinal data systems with data from other State agencies. Discussion: While the commenter noted several ways in which statewide longitudinal data systems could be expanded, we do not believe that it is necessary to include this information in the priority, nor to encourage States to link their longitudinal data systems with data from other agencies. How States expand their data systems will depend on the current needs, resources, and capabilities of each State's statewide longitudinal data system. We remind States that they must consider how to protect student privacy as data are shared across agencies. Successful applicants that receive Race to the Top grant awards will need to comply with the Family Educational Rights and Privacy Act (FERPA), including 34 CFR Part 99, as well as State and local requirements regarding privacy. Changes: None. Comment: Many commenters recommended that statewide longitudinal data systems include student-level data on transfers, chronic [[Page 59714]] absenteeism, and in- and out-of-school suspensions, as well as school dropout rates, dropout and re-enrollment data, and data on students completing P-16 programs. One commenter recommended that data on ``student mobility'' be included in all data gathering and reporting. Other commenters strongly recommended that State longitudinal data systems include measures of school safety, culture, and climate. Discussion: Applicants for Race to the Top grants will already be judged on the extent to which the State has a statewide longitudinal data system that includes all of the America COMPETES Act elements (see criterion (C)(1)). Those elements include, among other, student level enrollment, demographic, and program participation information; and student-level information about the points at which students exit, transfer in, transfer out, dropout, or complete P-16 education programs. It would not, therefore, be appropriate to include these elements in priority 4, which is focused on expanding statewide longitudinal data systems. However, we believe that it is appropriate to reference in priority 4 linking data from at-risk and dropout prevention programs, school climate and culture programs, and information on student mobility. Such data will complement and expand the data that States will be collecting through the America COMPETES Act elements. Therefore, we are adding language to the priority to refer to at-risk and dropout prevention programs, school climate and culture programs, and information on student mobility. For clarity, we also are adding a parenthetical following ``human resources.'' Changes: We have added the phrase ``at-risk and dropout prevention programs, and school climate and culture programs, as well as information on student mobility'' following ``early childhood programs'' in priority 4. We also have added ``(i.e., information on teachers, principals, and other staff)'' following ``human resources.'' Comment: None. Discussion: Throughout this notice, we have used the term ``English language learner,'' rather than ``limited English proficient,'' whenever possible. During our internal review, we noted that we inadvertently used ``limited English proficient'' in priority 4. Therefore, we are changing ``limited English proficient,'' to ``English language learner'' in priority 4. Changes: We have replaced ``limited English proficiency'' with ``English language learner'' in priority 4. Comment: Two commenters recommended that statewide longitudinal data systems include data on all postsecondary students, including adults who are enrolled part[hyphen]time, taking non[hyphen]credit courses, or participating in remedial programs. These commenters also recommended that statewide longitudinal data systems include data on participants in other educational and workforce training programs such as adult basic education programs. Several commenters recommended referencing data on career placements and State employment wage records as areas in which States should expand their systems. Discussion: As priority 4 already references postsecondary data, we do not believe it is necessary to add specific detail about the types of postsecondary data that States should collect. Nor do we believe that it is necessary to reference data on career placements and State employment wage records. States that believe such data are important to their overall reform strategy can certainly propose to expand their statewide longitudinal data base by adding these elements. Changes: None. Comment: Two commenters referred to the statement in the proposed priority stating that the Secretary was interested in applications in which States propose working together to adapt statewide longitudinal data systems, rather than having each State build such systems independently. The commenters requested guidance on how States should work together and asked for clarity about whether one State should be designated as the lead and what would happen if only one of the States in the partnership is successful in receiving a Race to the Top award. Discussion: States that propose to work together to adapt their statewide longitudinal data systems should include these proposed efforts in their reform plan and show how these efforts are coordinated with the State's larger reform efforts. When developing their plans, States should propose alternative options should one of the States not be awarded Race to the Top funds and be unable to devote other funds to achieve the outlined goals. Changes: None. Priority 5--Invitational Priority--P-20 Coordination, Vertical and Horizontal Alignment (Proposed Priority 4): Comment: Several commenters recommended that priority 5, regarding P-20 coordination, include an emphasis on aligning a State's educational system with other State agencies and community organizations. The commenters stated that such ``horizontal'' alignment is just as important as ``vertical alignment,'' particularly for high- need students. One commenter recommended that the Department require State reform plans to provide information about how all parts of the State's education system will work to improve student achievement and the overall quality of schools, and how the State's education system will work with other supporting agencies and institutions to address the needs of all students. The commenter also recommended that State reform plans address how the improvement process will be managed effectively both within the educational system and across supporting agencies and institutions. Numerous commenters stated that community-based organizations play a key role in assisting youth at the secondary level, particularly in helping them transition to postsecondary education, and therefore, should be included as partners in creating a seamless P-20 route for students. A few commenters stated that the educational system should work with child welfare, juvenile justice, and criminal justice agencies to help re-engage high school dropouts. Discussion: We agree that priority 5 would be strengthened by including a focus on coordinating educational systems with other State agencies and community organizations that provide services to students that are beyond the capacity of schools to provide. This would include, for example, community-based organizations that serve youth, as well as child welfare, juvenile justice, and criminal justice agencies, as mentioned by commenters. Therefore, we are revising the priority, as well as the title of the priority, to reflect a focus on the ``horizontal alignment'' of the educational system with other agencies and community organizations. Applicants that choose to address priority 5 should include in their State reform plans how all parts of the education system will coordinate their work to create a more seamless P-20 route for students--both vertically, to ensure that students exiting one level of the education system are prepared for success in the next, as well as horizontally, to ensure that services across schools, State agencies, and community partners are coordinated and aligned. With regard to the comment that State reform plans address how the improvement process will be managed effectively, we note that criterion (A)(2) focuses on the extent to which States have built strong statewide capacity to [[Page 59715]] implement, scale up, and sustain their proposed reform plans. Changes: We have changed the title of priority 5 to: P-20 Coordination, Vertical and Horizontal Alignment. In addition we have added ``and other State agencies and community partners (e.g., child welfare, juvenile justice, and criminal justice agencies)'' following ``organizations'' in the first sentence of the priority. Finally, we have added the following sentence at the end of the priority: ``Horizontal alignment, that is, coordination of services across schools, State agencies, and community partners, is also important to ensure that high-need students (as defined in this notice) have access to the broad array of opportunities and services they need and that are beyond the capacity of the school itself to provide.'' Comment: Many commenters recommended changing priority 5 from an invitational priority to a competitive preference priority, stating that P-20 alignment efforts are key to improving student transitions, and ultimately, student success. A few commenters recommended changing priority 5 from an invitational priority to an absolute priority. One commenter stated that coordination across and within systems can improve instruction, service delivery, and communication, and thus create an environment that encourages innovation. Discussion: We believe that priority 5 is appropriately designated as an invitational priority because it extends beyond the core K-12 focus of the Race to the Top program. States will already be judged on the extent to which they set forth a comprehensive and coherent reform agenda for improving student outcomes statewide (see criterion (A)(1)) and the extent to which they enlist strong statewide support and commitment for their plans from a broad group of stakeholders, which may include other State agencies, nonprofit organizations, and community-based organizations (see criterion (A)(2)(ii)). While we believe that the focus of priority 5 is important, it is not part of the core work that States must do to address the four education reform areas. Therefore, we decline to re-designate priority 5 as an absolute priority or a competitive preference priority. Changes: None. Comment: Several commenters recommended that priority 5 encourage collaboration between K-12 schools, higher education, and workforce development organizations in order to create pathways to college and work. One commenter stated that partnerships with workforce development organizations would add relevance to classroom instruction and help develop school-work partnerships. Discussion: We agree with the commenters and are changing ``workforce organizations'' to ``workforce development organizations'' to be clear that such organizations are important to creating a more seamless P-20 route for students. We also are including careers as an example of a critical transition point. Changes: We have changed ``workforce organizations'' to ``workforce development organizations.'' In the parenthetical following ``each point where a transition occurs,'' we have changed ``postsecondary'' to ``postsecondary/careers.'' Comment: Two commenters recommended including family engagement in each State's P-20 plan. Discussion: As part of its overall reform plan, States will be judged on the extent to which they have enlisted strong statewide support and commitment from a broad array of stakeholders, which includes community organizations, such as parent-teacher associations. Therefore, we do not believe it is necessary to add family engagement in this priority, as recommended by the commenters. We also note that priority 6 specifically focuses on flexibilities and autonomies for school-level reform, including those related to implementing strategies to effectively engage families and communities in supporting the academic success of their students (see paragraph (vii) in priority 6). Changes: None. Comment: One commenter recommended that the reference to vertical alignment in this priority include multiple education pathways to graduating from high school, such as alternative education programs, general educational development (GED) programs, and community college programs. Another commenter recommended that priority 5 focus on alignment between the traditional education system and alternative education programs for high school dropouts. Two commenters urged the Department to include adult education programs in this priority, stating that adult education programs play a key role in the P-20 route for some students, particularly English language learners. Discussion: Priority 5 refers to K-12 schools, postsecondary institutions, workforce development organizations, and other State agencies and community partners, which would encompass the programs referenced by the commenters. We do not believe that the notice needs to include additional references to these programs or to other specific types of schools or programs. Therefore, we decline to make the changes requested by the commenters. Changes: None. Comment: Many commenters highlighted the importance of improving the transition from early childhood to K-12 programs. One commenter asked that States be allowed to focus on coordination between early childhood and elementary school exclusively and without penalty for excluding middle school, high school, and post-secondary education in their plans. One commenter recommended that the Department more explicitly identify the ways in which early childhood and higher education sectors should participate in States' reform strategies and provide guidance on how cross-system alignment will be evaluated in the peer review process. Two commenters recommended that SEAs work with State early childhood advisory councils to improve the transition from early childhood programs to K-12 programs. Discussion: As discussed elsewhere, we are adding a new invitational priority 3 on improving early educational outcomes for high-need students who are young children, which includes a focus on improving transitions between preschool and kindergarten. With regard to the comment asking whether States could focus on the transition between early childhood and elementary school exclusively without penalty for excluding middle and high school transitions, and the comment regarding how alignment will be evaluated in the peer review process, we note that States will be judged on the extent to which their plans set forth comprehensive and coherent reform agendas for improving student outcomes statewide (see criterion (A)(1)), and on the extent to which States have enlisted strong statewide support and commitment for their plans from a broad group of stakeholders, which may include IHEs and agencies providing early childhood education (see criterion (A)(2)(ii)). States that choose to address priority 5 should discuss how to coordinate all parts of their systems to create more seamless P-20 routes for students--both vertically, to ensure that students exiting one level of the education system are prepared for success in the next, and horizontally, to ensure that services across schools, State agencies and community partners are coordinated and aligned. The ways in which early childhood and higher education programs participate in States' reform strategies [[Page 59716]] will vary from State to State depending on the needs and resources in each State. Therefore, we decline to include in priority 5 specific ways in which these sectors should participate in their State's reform plans, as requested by one commenter. We agree that one way to improve transitions from early childhood programs to K-12 programs is for SEAs to work with State early childhood advisory councils. We are not including specific examples of processes the State may use to improve transitions across the P-20 system; we believe such decisions are best left to local decision- makers. Changes: None. Comment: Two commenters recommended adding a reference in this priority to middle school transitions (i.e., elementary to middle school and middle to high school) because these transitions can be particularly challenging with the increased expectations for student performance and responsibility, often in environments that are far less personalized than elementary schools. Discussion: We agree that transitions to and from middle school can be challenging. Ensuring smooth transitions from elementary to middle school and from middle school to high school would be important aspects of creating a seamless P-20 route for students. The fact that priority 5 does not specifically reference the transitions to and from middle school does not mean that State reform plans should not include efforts to improve these important transitions. We note that the parenthetical in priority 5 provides examples of critical transition points before and after K-12 and is not meant to exclude transition points within K- 12 that States may address within their core Race to the Top reform plans. Changes: None. Comment: A few commenters requested that priority 5 include a requirement to coordinate early childhood programs that serve children from birth to age five. These commenters pointed to research documenting the importance of high quality education in the first three years of life. Discussion: We agree that the Race to the Top program should recognize the importance of early learning programs in preparing children for success in school. Therefore, as noted elsewhere, we are adding priority 3 to focus on improving early educational outcomes for high-need students who are young children (pre-kindergarten through third grade). Because Race to the Top focuses its efforts primarily on States and LEAs, an early childhood educational focus starting in pre- kindergarten seems most applicable. The Department has other programs that will focus exclusively and comprehensively on children younger than pre-kindergarten age. Changes: None. Comment: One commenter recommended that States include private schools in developing their plans to create a more seamless P-20 route for students. The commenter noted that many students attend both public and private schools at various times in their educational careers. Discussion: There is nothing that would preclude a State from including in its plan efforts to improve coordination with private schools. We note that nothing in the Race to the Top program requires a State that receives funds under Race to the Top to include private schools in the four reform areas. Because the Race to the Top program is directed to improving public K-12 education, we decline to include a reference to private schools in priority 5, which addresses a more seamless P-20 route for students. Changes: None. Comment: One commenter asked whether the focus of priority 5 is on developing a P-20 data system. Another commenter asked how the data elements in a P-20 system would differ from a P-16 system's required elements. Discussion: Priority 5 focuses on improving all parts of the education system by coordinating within the educational system (e.g., between early childhood programs, K-12 schools, postsecondary institutions) and between the educational system and other State agencies and community partners (e.g., child welfare, juvenile justice, and criminal justice agencies). Priority 5 is not focused on P-20 data systems; that is the focus of priority 4, Expansion and Adaptation of Statewide Longitudinal Data Systems. Under criterion (C)(1), States will be judged on the extent to which they have a statewide longitudinal data system that includes the America COMPETES Act elements. Beyond these 12 elements, the Department has not specified any additional elements that States must include in their statewide longitudinal data systems. Changes: None. Comment: One commenter recommended that States use longitudinal data to evaluate and improve the effectiveness of programs designed to facilitate vertical alignment in the education system. Two commenters recommended that the Department include an incentive in this priority for States and LEAs to learn from LEAs with outstanding records in data development and reporting in order to improve the vertical alignment of the State's education system. Discussion: We agree that longitudinal data could be used to evaluate and improve the effectiveness of programs designed to improve transitions from one level of the education system to another. We also agree that States and LEAs should learn from each other on using data to improve the vertical alignment of educational systems. Priorities 3, 4, and 5 encourage States to undertake such practices. We note that States receiving Race to the Top funds, along with their LEAs and schools, are expected to identify and share promising practices, make work freely available within and across States, make data available in appropriate ways to stakeholders and researchers, and help all States focus on continuous improvement of student outcomes. Changes: None. Priority 6--Invitational Priority--School-Level Conditions for Reform, Innovation, and Learning (Proposed Priority 5). General: Comment: Numerous commenters expressed support for priority 6. While some commenters stated that it was appropriate for priority 6 to be an invitational priority, numerous other commenters recommended changing priority 6 to a competitive preference priority stating that the conditions listed for reform and innovation are critical to supporting school reform efforts. One commenter stated that it is important to give priority to school-level conditions for reform because reform is most evident when changes are implemented at the local level, where student learning can be directly and immediately influenced. Several commenters urged the Department to make priority 6 a competitive preference priority in order to ensure that districts create the preconditions for dramatically improving student achievement. Other commenters stated that the flexibilities and autonomies listed in the priority are essential to school success and that it is highly unlikely that any State will turn around low- performing schools without these ingredients. Another commenter stated that LEA actions are fundamental to enabling schools to turn around and that if this priority was a competitive preference priority, it would motivate LEAs to undertake challenging reforms. Lastly, one commenter recommended that the priority be changed to an absolute priority. Discussion: States may choose to address priority 6, which examines the [[Page 59717]] extent to which a State's participating LEAs are broadly creating the conditions for reform and innovation by providing schools with flexibilities and autonomies. All States, however, will be rewarded for flexibilities and autonomies that are provided to schools in the highest need situations--turning around persistently lowest-achieving schools--as part of criterion (E)(2). In addition, criterion (F)(2) will assess the extent to which States ensure successful conditions for high-performing charter schools and other innovative schools. Therefore, we do not believe it is necessary to change priority 6 to an absolute or competitive preference priority. Changes: None. Comment: A few commenters noted that priority 6 focuses on school- level conditions for reform and innovation but does not speak to the conditions that are necessary for student learning. The commenters recommended that the title and content of the priority be changed to also focus on creating the school-level conditions for learning. One commenter stated that school-level conditions for reform should be clearly defined in the notice to ensure that all of the comprehensive learning opportunities necessary for school success are in place. Discussion: We agree with the commenters that priority 6 should emphasize reform and innovation in the service of learning, and thus are adding ``learning'' to the title of the priority. We also are clarifying, in the text of the priority, that the Secretary is interested in applications in which the State's participating LEAs create the conditions for reform and innovation, as well as the conditions for learning. We decline to provide an exhaustive list of school-level conditions for reform as requested by one commenter as such conditions will vary depending on the unique needs of schools and communities. Therefore, priority 6 only includes examples of flexibilities and autonomies that an LEA might provide to its schools in order to help create the conditions for reform, innovation, and learning. We also are making a few technical edits for clarity. Changes: We have changed the title of priority 6 to ``School-Level Conditions for Reform, Innovation, and Learning.'' We have added the phrase ``seek to create the conditions for reform and innovation as well as the conditions for learning. * * *'' following ``The Secretary is particularly interested in applications in which the State's participating LEAs.'' Comment: One commenter stated that in order to meet priority 6, States should describe the ways in which their participating LEAs provide schools, in particular turnaround schools, with flexibilities and autonomies conducive to reform and innovation. Discussion: Under criterion (E)(2), States must describe the ways in which they will support their LEAs to implement the flexibilities provided in the school intervention models (described in Appendix C) for their persistently lowest-achieving schools. Therefore, in addressing priority 6, a State should describe other flexibilities and autonomies that its LEAs currently provide, or plan to provide, to their schools in order to create the conditions for reform, innovation, and learning. Changes: None. Comment: One commenter recommended that priority 6 be changed to reach beyond LEA-school governance to include State-LEA flexibility and autonomy. The commenter stated that emphasis should be placed on demonstrating how changes in governance and rules affect school reform efforts and instructional innovations. The commenter further recommended that we add examples of flexibilities and autonomies conducive to reform and innovation such as coordinated planning between categorical programs and budgets, changing education delivery models to increase productivity, and more efficiently using existing learning time and resources. A few commenters recommended that the Department provide additional regulatory waivers and flexibilities to improve the coordination of funds and create the conditions for systemic reforms and instructional innovations. One commenter stated that Federal funding and regulatory flexibility could have a significant effect on State and LEA reform efforts and suggested that funds be competitively awarded in return for a State meeting a number of key requirements. Discussion: The Department is placing particular emphasis on these school-level flexibilities because their effectiveness has been shown in a number of educational settings and because they are related to efforts to turn around struggling schools, which is a priority of the ARRA. We are, however, open to State innovation around exploring further flexibilities with their LEAs and, to the extent that such flexibilities are in place, the State could describe them in response to criterion (F)(3), Demonstrating Other Significant Reform Conditions. We also note that under criterion (A)(2)(i)(d), a State will be evaluated based on its capacity to accomplish its plan and targets by coordinating, reallocating, or repurposing education funds from other Federal, State, and local sources where feasible. We, therefore, believe it is unnecessary to add to priority 6 the language regarding coordinated planning between categorical programs and budgets and changing delivery models suggested by the commenter. In response to commenters who recommended that the Department provide additional regulatory waivers and flexibilities, we note that such waivers and flexibilities are often limited by statute. However, the Department fully supports efforts to coordinate the use of funds in order to make the most efficient and effective use of limited resources and will continue to consider States' requests for waivers that are permissible under current Federal statutes and regulations. Changes: None. Comment: A few commenters recommended that the list of flexibilities and autonomies conducive to reform and innovation include providing high-quality, engaging curricula and instruction that focus on real-world problem solving. The commenters also recommended that instruction be consistent with the principles of universal design for learning. Discussion: Several Race to the Top selection criteria established in this final notice emphasize an approach to curriculum and instruction that is based on an evidence-driven cycle of continuous instructional improvement (see criteria (B)(3), (C)(3), and (D)(5)). Because this issue is addressed directly in the criteria, we do not believe it is necessary to reference specific principles used to design curricula or instruction (i.e., universal design for learning). Changes: None. Comment: A few commenters requested that priority 6 clearly state that the flexibilities and autonomies provided to schools must not include waiving the program requirements under the IDEA. Discussion: There is nothing in priority 6 to suggest that LEAs would be permitted to waive program requirements required under other Federal laws and regulations, including those required by the IDEA. Therefore, we believe it is unnecessary to add the language requested by the commenters. Changes: None. Comment: One commenter requested that the final notice provide examples of flexibilities and autonomies that LEAs could provide to schools to improve early learning. The commenter provided numerous examples, including [[Page 59718]] increasing the use of Title I funds for early learning programs and permitting the use of school facilities for early learning programs and family centers. Discussion: Several of the flexibilities and autonomies included in priority 6 are applicable to early learning--for example, flexibility in selecting staff (paragraph (i)) and controlling the school's budget (paragraph (iii)). Therefore, we do not believe that examples specifically applicable to early learning are necessary. We note that, as discussed elsewhere in this notice, we are adding an invitational priority (Priority 3) focused on early learning. An applicant who chooses to address the early childhood priority could choose to include flexibilities, such as those recommended by the commenter, in its application. Changes: None. Comment: Numerous commenters recommended that the list of flexibilities and autonomies conducive to reform and innovation include charter schools and charter school autonomies. Several of these commenters recommended that States be rewarded for their past and proposed efforts to support charter school flexibilities and, conversely, that States should lose points if they do not provide adequate school-level autonomy or are implementing efforts to restrict charter school flexibility. One commenter suggested that we clarify that flexibilities and autonomies conducive to reform and innovation do not include policies that would exempt charter schools or other non- traditional public schools from open enrollment mandates or from requirements that they be subject to and rated by the same academic achievement standards as traditional public schools. Discussion: As part of its application, a State is already asked to address several criteria to ensure that it is creating the conditions for high-quality charter schools. (See criterion (F)(2)). Therefore, we decline to include additional criteria related to charter schools in priority 6. We also decline to add language specifying the flexibilities and autonomies that LEAs may provide to charter schools. State and local governments possess the authority to authorize charter schools and as such, requirements for charter school admissions are primarily State and local matters. Changes: None. Selecting Staff (Paragraph (i)) Comment: One commenter recommended that paragraph (i) of this priority specifically refer to schools having the flexibility to select ``leadership team members.'' Another commenter stated that school principals must have the authority to replace consistently low- performing educators and suggested changing paragraph (i) to clarify that principals should be given the authority to select and replace staff. Discussion: We decline to add ``leadership team members'' to paragraph (i) in priority 6 because we are unsure to whom the term refers. With regard to the suggestion that we refer specifically to principals selecting and replacing staff, we note that there may be other school leaders or groups of school staff responsible for hiring staff (e.g., department chairs; a panel of teachers, parents, and the principal; an executive in a private management organization). Therefore, we decline to make the change proposed by the commenters. Changes: None. Increased Learning Time (Paragraph (ii)) Comment: Many commenters expressed support for reform efforts that put in place new structures and formats for the school day or year in order to expand learning time. Commenters provided many examples of activities that should be conducted during expanded learning time including extra-curricular pursuits, experiential learning, enrichment activities, family and community engagement, recreational activities, and activities that support students' transition between grade levels. Other commenters focused on the use of expanded learning time for academic supports, and as a strategy to improve student achievement, close achievement gaps, and support struggling schools. One commenter stated that priority 6 should include other flexibilities such as expanding opportunities for youth that include, but are not limited to, a longer school day. Several commenters recommended clarifying that expanded learning time includes after-school and summer school programs. Another commenter strongly recommended that the final notice clarify that expanded learning time includes strategies that go beyond those that mirror the instruction provided to students during the school day. Other commenters stated that it is important for the Department to acknowledge that expanded learning time includes increasing educators' learning time for activities such as professional development that is collaborative, on-site, and tailored to the needs of school staff and leadership, and to allow teachers to plan and learn together. Discussion: We appreciate the numerous comments we received on increasing learning time. We acknowledge that the term, ``expanded learning time'' is typically used to refer to programs that redesign the school day, week, and year to provide additional hours of learning time, and that ``extended learning time'' is typically used to describe before school, after school, and summer programs. We, therefore, are defining a new term, increased learning time, to indicate the need for schools to provide additional time for academic work to improve the proficiency of students in core academic subjects, as well as for additional subjects and enrichment activities that can contribute to a well-rounded education. We agree with commenters that teachers could also use the additional time to collaborate, plan, and engage in professional development. Changes: We have replaced ``expanded learning time'' with ``increased learning time.'' We also have added a definition of increased learning time in the definitions section of this notice to read as follows: ``Increased learning time means using a longer school day, week, or year schedule to significantly increase the total number of school hours to include additional time for (a) instruction in core academic subjects, including English; reading or language arts; mathematics; science; foreign languages; civics and government; economics; arts; history; and geography; (b) instruction in other subjects and enrichment activities that contribute to a well-rounded education, including, for example, physical education, service learning, and experiential and work-based learning opportunities that are provided by partnering, as appropriate, with other organizations; and (c) teachers to collaborate, plan, and engage in professional development within and across grades and subjects.'' \2\ --------------------------------------------------------------------------- \2\ Research supports the effectiveness of well-designed programs that expand learning time by a minimum of 300 hours per school year. (See Frazier, Julie A.; Morrison, Frederick J. ``The Influence of Extended-year Schooling on Growth of Achievement and Perceived Competence in Early Elementary School.'' Child Development. Vol. 69 (2), April 1998, pp.495-497 and research done by Mass2020.) Extending learning into before- and after-school hours can be difficult to implement effectively, but is permissible under this definition with encouragement to closely integrate and coordinate academic work between in-school and out-of school. (See James-Burdumy, Susanne; Dynarski, Mark; Deke, John. ``When Elementary Schools Stay Open Late: Results from The National Evaluation of the 21st Century Community Learning Centers Program.'' http://www.mathematica-mpr.com/publications/redirect_ PubsDB.asp?strSite=http://epa.sagepub.com/cgi/content/abstract/29/4/ 296 Educational Evaluation and Policy Analysis, Vol. 29 (4), December 2007, Document No. PP07-121.) --------------------------------------------------------------------------- [[Page 59719]] Comment: Many commenters recommended that priority 6 focus on removing barriers to innovative approaches to serving students in after-school and summer school programs. The commenters stated that schools should be encouraged to allow the use of school buildings for summer programs. Other commenters recommended requiring LEAs to coordinate funding streams for after-school and summer school programs, such as those tied to Title I, 21st Century Community Learning Centers, and other Federal, State, and local funds in order to maximize impact, improve efficiencies, and provide comprehensive services. Discussion: Priority 6 focuses on creating the conditions for reform, innovation, and learning at the school level and includes a list of the types of flexibility and autonomy that LEAs may provide to schools; the list provides examples and is not exhaustive. We do not believe it is necessary to include the very specific flexibility of removing barriers to using school buildings for after-school and summer school programs. Likewise, flexibilities that permit coordinating funding streams for after-school and summer school programs are already covered in paragraph (iii) of the priority, which references placing budgets under the school's control. Changes: None. Comment: One commenter recommended that LEAs be encouraged to form partnerships with providers of out-of-school-time programming that have proven outcomes and that can bring innovative approaches to support true reform. Another commenter recommended that States ensure that nonprofit partners have the opportunity to apply for extended learning funds in partnership with one or more struggling schools in order to maximize competition and increase the quality of programs provided. One commenter recommended requiring States to ensure that expanded learning time models do not limit staffing to existing teachers. The commenter stated that flexibility should be provided to engage educators outside of the school such as tutors, mentors, individuals in teaching fellowship programs and alternative certification programs, and volunteers from the community, business, and industry. Discussion: Developing local partnerships can be an effective strategy to move local school reform agendas forward, particularly in providing comprehensive services to high-need students. However, we believe it would be inappropriate to require States to form partnerships with nonprofit organizations or individuals outside of the school; such decisions are best left to local decision-makers who understand the unique needs of their schools and the resources available in their communities. We are changing the language in paragraph (v) regarding comprehensive services to high-need students to include examples of how such services might be provided to high-need students. Changes: The parenthetical in paragraph (v) now reads, ``(e.g., by mentors and other caring adults; through local partnerships with community-based organizations, nonprofit organizations, and other providers).'' Comment: One commenter supported expanded learning time but stated that educators should not be forced to work longer hours for the same compensation and that adjustments to work schedules should be determined locally between the district and educators and bargained where collective bargaining agreements exist. A few commenters stated that collaboration among labor, management, and parents is critical for expanded learning time models to succeed. Discussion: Decisions about work hours and compensation are determined at the local level. As with all educational reform efforts, we believe that collaboration among stakeholders is critical to success. Changes: None. Comment: One commenter recommended that the final notice provide a clear picture of how strategies for expanded learning time and comprehensive services for high-need students fit together as part of a broader approach to reform and recommended that language be added to encourage applications that demonstrate how States and LEAs will align their strategies to produce results. Discussion: It will be up to each applicant to describe how its plan for reform is comprehensive and coherent and will increase student achievement, reduce achievement gaps, and increase graduation rates. Absolute priority 1 specifically requires that States comprehensively address each of the four education reform areas specified in the ARRA and demonstrate that the State and its participating LEAs are taking a systemic approach to education reform. Applicants who choose to address priority 6 should address how their approach to meeting this priority fits into the State's overall reform efforts. Changes: None. Budgets (Paragraph (iii)) Comment: One commenter recommended revising paragraph (iii) regarding placing budgets under the schools' control to ensure that teachers and parents are involved in making budget decisions. Discussion: The process that a school or LEA uses to establish its budget is a local matter. Therefore, we decline to add the language requested by the commenter. Changes: None. Credit Based on Student Performance (Paragraph (iv)) Comment: Several commenters expressed support for awarding credit to students based on student performance instead of instructional time and providing multiple pathways to a graduation with a regular high school diploma. One commenter recommended that funds be used to encourage State policies that allow middle or high school students to receive high school graduation credit or to meet a subject area requirement earlier than typically would be expected. The commenters advocated for options that create flexibility for students without sacrificing rigorous learning and cited school-work partnerships, diploma-plus programs, and dual enrollment (high school-community college) programs as examples of innovative approaches to creating multiple options that help students graduate from high school and pursue additional educational goals. Discussion: We believe that the commenters' recommendations are all addressed in paragraph (iv), which provides for ``awarding credit to students based on student performance instead of instructional time.'' We, therefore, do not see a need to add the commenter's recommended language in priority 6. Changes: None. Comprehensive Services (Paragraph (v)) Comment: A few commenters noted that instruction and services for high-need students cannot be provided by traditional education systems alone and recommended adding language to the priority to emphasize the importance of community-based organizations and nonprofit organizations in providing comprehensive services to high-need students. One commenter stated that the final notice should clarify that the goal of State and local educational agencies should be to build a comprehensive picture of children's progress--academically, socially, and in terms of health and well-being. One commenter stated that in order to provide [[Page 59720]] comprehensive services to high-need students, States must create a safety net of wrap-around services designed to increase student success and focus on both community- and district-level conditions. Another commenter suggested using the term ``comprehensive supports'' rather than ``comprehensive services,'' stating that ``comprehensive supports'' includes services and has more salience with educators. Another commenter recommended clarifying that comprehensive services for high-need students address the health, safety, social, emotional, behavioral, physical, and educational needs of a child. Discussion: We agree with the commenters that high-need students often require a broad array of services that are beyond the capacity of the school itself to provide, and that community-based organizations and nonprofit organizations play an important role in meeting these needs. As noted in an earlier comment regarding the role of community- based organizations and nonprofit organizations in schools that provide increased learning time, we are changing paragraph (v) to reference community-based organizations and nonprofit organizations. With regard to comments concerning the need for comprehensive services and creating a safety net of wrap-around services with involvement of both communities and districts, we note that priority 5 focuses on the need to coordinate services across schools, State agencies, and community partners in order to ensure that high-need students have access to the broad array of opportunities and services they need (see the discussion on priority 5). We decline to change the term ``comprehensive services'' to ``comprehensive supports,'' as requested by one commenter; we do not agree that the two terms are substantively different or that one term has more salience for educators than the other. We also decline to specify the array of services included in ``comprehensive services'' because, by doing so, we could inadvertently restrict the range of services that a State may determine are necessary to serve high-need students. Changes: None. II. Requirements Eligibility Requirements Eligibility Requirement (a): State Fiscal Stabilization Fund (Stabilization) Phase 1 and 2: Comment: Many commenters expressed support for the eligibility requirement that States have their State Fiscal Stabilization Fund program Phase 1 and Phase 2 applications approved in order to be eligible for a Race to the Top award. Other commenters expressed concern that States may have difficulty obtaining approval of their Stabilization Phase 2 applications in time to submit a Race to the Top application. One commenter expressed concern that the Department's approval of Stabilization Phase 2 applications may occur too late for a State to apply during Phase 1 of the Race to the Top competition. One commenter specifically noted the difficulty in satisfying the data requirements for Stabilization Phase 2 in time to apply for the Race to the Top competition. Some commenters requested information pertaining to the timing of Stabilization Phase 2 applications and the Race to the Top competition. Discussion: The eligibility requirement pertaining to the approval of Stabilization applications is being changed to require only that the State have approved Stabilization Phase 1 and Phase 2 applications by the time the State is awarded a Race to the Top grant. Thus, a State's Stabilization Phase 2 application will not need to be approved at the time it prepares or submits its Race to the Top application. Changes: Eligibility requirement (a) has been changed to read: ``A State must meet the following requirements in order to be eligible to receive funds under this program. (a) The State's applications for funding under Phase 1 and Phase 2 of the State Fiscal Stabilization Fund program must be approved by the Department prior to the State being awarded a Race to the Top grant.'' Eligibility Requirement (b): Linking Student Data to Teachers and Principals: Comment: Numerous commenters expressed their support for evaluating teachers and principals based on student achievement or growth. These commenters suggested that the final notice should require States to use student growth data in teacher and principal evaluations. Several commenters offered their support for the requirement that a State not have any barriers to linking student achievement or student growth data to teacher and principal evaluations. These commenters specifically noted that teachers should be judged by their effectiveness, not by their credentials or years of service. Several commenters, however, claimed that there is a lack of research or evidence demonstrating that the use of such data for teacher and principal evaluations has any positive impact on teacher, principal, or student performance. A few commenters disagreed with the Department's reference to research indicating that teacher qualifications, including certification status and years of experience, are not accurately predictive of teacher quality. Other commenters identified research explaining the difficulty in disaggregating student achievement data to determine a teacher's effect from other variables. One commenter suggested that States should pass laws requiring a peer reviewed validation of any value-added methodology before including student achievement data as part of any evaluation or compensation mechanism and further argued that such laws should not constitute a State barrier under the eligibility requirements. Discussion: As indicated in the NPP, we believe that research clearly shows that teacher and principal quality are critical contributors to student learning. The Department believes that student achievement and student growth data are meaningful measures of teacher and principal effectiveness, and therefore, should be considered as a part of a rigorous, transparent and fair evaluation system. Consequently, legal barriers to linking data about student achievement or student growth to teachers and principals for evaluation purposes effectively prevents schools from having the core information systems they need to serve students well. For these reasons, we decline to make substantive changes to eligibility requirement (b). Changes: None. Comment: Several commenters asked whether teacher or principal contracts or local collective bargaining agreements that prohibit the use of student achievement data for teacher and principal evaluations would constitute a State barrier, thus making a State ineligible for the Race to the Top competition. One commenter noted that one specific State lacks control over teacher and principal evaluation systems. Discussion: The Department has revised eligibility requirement (b) to clarify that the State must not have any legal, statutory, or regulatory barriers at the State level to linking student achievement or student growth data to teachers and principals for purposes of evaluation. Therefore, a State would be eligible to apply for a Race to the Top grant even if a teacher or principal contract or collective bargaining agreement at the local level prohibited the use of student achievement or student growth data for evaluation purposes. [[Page 59721]] Changes: Eligibility requirement (b) has been changed to read: ``At the time the State submits its application, there must not be any legal, statutory, or regulatory barriers at the State level to linking data on student achievement (as defined in this notice) or student growth (as defined in this notice) to teachers and principals for the purpose of teacher and principal evaluation.'' Comment: One commenter suggested limiting the eligibility requirements pertaining to linking student achievement data to teacher and principal evaluations to exclude educators working in early learning or child care programs. This commenter claimed that teacher and principal evaluation systems would not be applicable to a State's proposal emphasizing early learning initiatives. Discussion: The Department believes that student growth data are strong measures of teacher effectiveness across the spectrum from preschool to grade 12. While traditional student achievement and student growth data may not be routinely collected in early learning settings, relevant student achievement and student growth data are available in other forms. Child outcome data should not be the only measures of teacher effectiveness in early learning settings, but can provide useful information to improve the effectiveness of early childhood educators and administrators when coupled with other quantitative and qualitative indicators. Changes: None. Comment: One commenter recommended that the notice clarify what level of change to a State law regarding linking data on student achievement or student growth to teachers and principals would be necessary in order to be eligible for Race to the Top funds. For example, one commenter asked if legislation to remove a barrier to linking student achievement data to teachers and principals would need to be enacted prior to applying for Race to the Top funds or whether the introduction of such legislation would be adequate to the meet eligibility requirements. Another commenter asked whether a State would need to enact legislation adopting its plan in its State education code to be eligible to apply for Race to the Top funds. Discussion: Eligibility requirement (b) contemplates only existing laws; a State will not be able to establish its eligibility based on intent to change those laws. There is no requirement in the ARRA or in this notice requiring States to enact legislation adopting their Race to the Top plans. Changes: None. Comment: Some commenters suggested that States should be eligible for the Race to the Top competition even if barriers exist to linking student achievement or student growth data to teachers and principals for evaluation purposes, so long as the State's reform plan only includes LEAs and charter schools that allow such linkages. One commenter argued that the eligibility requirement is unfair because LEAs without such prohibitions would not receive Race to the Top funds if they were situated in a State with such barriers. Discussion: Under eligibility requirement (b), States are required to demonstrate that they do not have any legal, statutory, or regulatory barriers at the State level to linking student achievement or student growth data to teachers and principals for the purpose of evaluations. States that have such barriers are not eligible for Race to the Top awards. Race to the Top is meant to provide an incentive for statewide reform and improvements, and is a competitive grant program encouraging States to be bold and innovative. While individual LEAs and charter schools in States with barriers may be ready and eager to use student growth data to identify and improve teacher and principal effectiveness, Race to the Top focuses on the extent to which the State's conditions and plans lead to statewide impact. Changes: None. Comment: Numerous commenters argued that one specific State's law, which prohibits linking teacher and student achievement data, should not disqualify it from applying for the Race to the Top competition. Some of these commenters argued that the State's law does not prohibit data linking between students and teachers at the district level where personnel decisions are made, and therefore should not make the State ineligible for Race to the Top funds. One commenter, however, specifically stated their support for the data linkage eligibility requirement with respect to the State. Another commenter argued that an existing statute regulating the use of student achievement data in tenure determinations in another State should not make the State ineligible to apply for the Race to the Top competition. The commenter argued that the statute does not prohibit use of student test data in annual teacher performance reviews or for tenure consideration. Discussion: As stated earlier, the Department believes that student growth should be one significant measure of several when evaluating teacher and principal effectiveness. State level data linkage barriers unduly restrict schools and LEAs from using student achievement or student growth data to identify and improve teacher and principal effectiveness. The Department also believes that schools and LEAs should have the ability to choose to use student achievement and student growth data in this manner. For this reason, the Department declines to exempt any one State from this requirement and encourages States to lift legal, statutory, and regulatory barriers that prohibit these linkages. The Department notes that this notice requires the State's Attorney General to certify that the State has no legal, statutory, or regulatory barriers at the State level to linking student achievement or student growth data to teachers and principals for the purpose of evaluations. Changes: None. Eligibility Overall Comment: Multiple commenters suggested adding an eligibility requirement to limit eligibility for Race to the Top funds to States that meet the requirements in their FY 2007 Annual Performance Report under the IDEA. Those commenters noted that States unable to meet basic IDEA requirements should not be eligible to apply for Race to the Top funds. Discussion: Race to the Top is a competitive grant program intended to improve educational outcomes for all students. The Department already has a mechanism to monitor States' progress, as reported in their Annual Performance Reports, in meeting the targets in their State Performance Plan under the IDEA. Therefore, we decline to include the requirement suggested by the commenter as an eligibility requirement in the Race to the Top competition. Changes: None. Comment: One commenter suggested the Department consider the number of outstanding audits and audit exceptions against a State for any Federal education program as part of the Race to the Top program eligibility determination. One commenter suggested that if awards were given to States with audit exceptions, conditions should be imposed on the award of funds, including onsite monitoring. Discussion: The Department has taken extraordinary measures to ensure accountability in the use of all ARRA funds, including the Race to the Top program, so that all dollars are used wisely and accounted for in a transparent manner. Indeed, as explained in the Reporting section of this final notice and the notice inviting [[Page 59722]] applications, successful applicants must comply with the ARRA annual reporting requirements in section 14008 of the ARRA and quarterly reporting requirements in section 1512(c) of the ARRA, which are designed to ensure thorough and public oversight of the expenditure of ARRA funds. The Department has established a Recovery Act Web site and hotline for members of the public to report suspected misuse of funds. Additionally, the Department has other mechanisms and protections in place to enforce and monitor progress and resolution of any prior audit findings from other programs. Accordingly, we do not believe it is necessary to add requirements pertaining to States that have audit exceptions. Changes: None. Application Requirements Reorganization of the Application Requirements Comment: None. Discussion: In order to streamline the application requirements and the criteria and reduce burden for applicants, we are removing from this final notice proposed application requirements that were duplicative of the criteria. The remaining application requirements are being renumbered, accordingly. For instance, proposed application requirement (c) concerning the level of State funding for education is being removed from the final application requirements but is still being retained in criterion (F)(1)(i); and proposed application requirement (d) concerning support from stakeholders is being removed but is still being retained in criterion (A)(2)(ii). In addition, we are revising the application requirements to make minor editorial changes, providing internal cross references to relevant portions of the notice, and reorganizing application requirement (e) to better clarify the components of this requirement. Changes: We have removed proposed application requirements (c) and (d). We have reordered the application requirements accordingly. We have made minor editorial changes to provide better clarification to this section, have clarified that the Governor must sign the assurances in Section IV of the application, and have reorganized application requirement (e). Comment: Some commenters recommended providing benchmarks or statutory tests to help provide consistency in how State Attorneys General determine and certify their State's eligibility for Race to the Top. Some commenters suggested that the Department provide a ``test'' for Attorneys General to apply to their State law to determine eligibility. Discussion: Under application requirement (f) (proposed application requirement (h)), the State's Attorney General is asked to certify that the State has no legal, statutory or regulatory barriers at the State level with respect to eligibility requirement (b). We interpret this to mean State constitutions, case law, statutes, or regulations. Interpretation of a State's laws falls uniquely within the expertise of the State Attorney General and therefore, we leave this task to the Attorney General. The Department notes that the certification requirement does not seek a formal legal opinion. Instead, the Department provides forms in the application for Attorneys General to sign certifying that (a) the description of, and statements and conclusions in the application concerning State law, statute, and regulation in its application are complete, accurate, and constitute a reasonable interpretation of State law, statute and regulation; and (b) that the State does not have any legal, statutory, or regulatory barriers at the State level to linking data on student achievement or student growth to teachers and principals for the purpose of teacher and principal evaluations. The certification of the Attorney General addresses this requirement. The applicant may provide explanatory information, if necessary. In addition, we note that we are changing application requirement (f) to be consistent with the changes to eligibility requirement (b), as discussed earlier, and separating application requirement (f) into two subparagraphs. Changes: Application requirement (f) has been made consistent with eligibility requirement (b), as discussed earlier, and separated into two subparagraphs. High-Need LEAs Comment: Many commenters had difficulty interpreting proposed application requirement (e)(2) that would have required States to explain in their budget plans how it will use Race to the Top funds to give priority to high-need LEAs over and above the participating LEA share. Discussion: First, the Department notes that it inadvertently neglected to use the statutory definition of high-need LEA in the NPP, as found in section 14013(2) of the ARRA. Accordingly, and as discussed in this notice, we are changing the definition of high-need LEA to reflect the statutory definition: ``[an LEA] that serves not fewer than 10,000 children from families with incomes below the poverty line; or for which not less than 20 percent of the children served by the LEA are from families with incomes below the poverty line.'' Consistent with section 14006(c) of the ARRA, States must subgrant 50 percent of their grant awards to participating LEAs, based on the LEAs' relative share of Title I, Part A allocations in the most recent year. We have clarified in application requirement (c)(2) that, because all Race to the Top grants will be made in 2010, relative shares will be based on total funding received in FY 2009, including both the regular Title I, Part A appropriation and the amount made available by the ARRA. Consistent with section 14005(c)(4) of the ARRA, application requirement (c)(2) requires a State to include in its application a budget detailing how the State will use Race to the Top funds to ``give priority to high-need LEAs'' beyond the base amount provided to all participating LEAs. States have flexibility to determine the meaning of ``give priority to,'' which could include, for example, additional funding, more comprehensive technical assistance, coordination of State or local social services for students in such LEAs, expanded professional development, and larger incentives for teachers and principals who agree to work in these LEAs. Changes: Application requirement (c)(2) has been revised to include: ``(Note: Because all Race to the Top grants will be made in 2010, relative shares will be based on total funding received in FY 2009, including both the regular Title I, Part A appropriation and the amount made available by the ARRA).'' Reporting Requirements Comment: Several commenters raised questions concerning accountability for Race to the Top funds. One commenter praised the proposed requirements but wanted greater detail on how we would ensure ``successful on-the-ground implementation'' of the Race to the Top program. One strategy suggested by the commenter was to withhold funds from States that do not meet the commitments they make in their Race to the Top applications. Other commenters recommended that Race to the Top funds be conditioned on meeting performance goals as reflected in the annual reports, or that the Department withhold funds from those States not meeting their commitments. Two commenters requested flexibility for States to revise their State plans to encourage continuous improvement. [[Page 59723]] Discussion: The Reporting Requirements section in this final notice explains that the Department plans to both support and carefully monitor State and LEA progress in meeting their goals, timelines, budgets, and annual performance targets. If we determine that a State is not meeting one or more of the requirements for this program, the Department may take a range of actions to remedy the situation, including placing the State in high-risk status, putting the State on reimbursement payment status, or delaying or withholding funds. The Department also recognizes that States may wish to, or need to, revise their Race to the Top plans occasionally to take into account changing circumstances; such revisions will be subject to approval by the Secretary. The Department recognizes that many of the accountability requirements of the Race to the Top program differ from those of the ESEA, and that winning States will be adding a new layer of goal- setting, performance measurement, and data collection to their existing accountability systems. Finally, to provide greater clarity and completeness to the Reporting Requirements section, we are including the reporting requirements contained in sections 1512(c) and 14008 of the ARRA. Changes: We have added the reporting requirements contained in sections 1512(c) and 14008 of the ARRA. Comment: One commenter argued that the Department may not use written performance agreements or cooperative agreements to monitor a State's progress because, they claimed, ARRA only allows grants monitoring. Another commenter stated that the Department should be a full participant in the Race to the Top program and, therefore, that Race to the Top awards should be cooperative agreements, rather than grants. Discussion: The Department intends to support States and LEAs through technical assistance, evaluations, and other mechanisms to facilitate them in meeting their goals, timelines, budgets, and annual performance targets. Contrary to the assertion by one commenter, the Department has the authority under the Federal Grant and Cooperative Agreement Act of 1977 (31 U.S.C. Chapter 63) to use written performance agreements or cooperative agreements to monitor Race to the Top grantee performance. As stated in the NPP and reiterated in this notice, the Department may require grantees to enter into a written performance or cooperative agreement with the Department as a condition of receiving the grant; a final determination will be made at the time of grant awards. We do not believe it is necessary to arbitrarily require these agreements for all grantees because the determination whether to use a cooperative agreement as the award instrument is based on the nature of the relationship and the activities to be performed by the grantee, and is therefore highly case specific. Changes: None. Program Requirements Evaluation Comment: In response to the NPP's request for advice on the best way to conduct an evaluation of the Race to the Top program, many commenters recommended that States conduct their own Race to the Top evaluations. These commenters believed that the likely breadth of variation in Race to the Top plans would make it difficult to conduct a national evaluation, and that State-level evaluations would provide the kind of detailed feedback needed to support continuous improvement. However, another commenter asserted that a relatively small number of States were expected to receive a Race to the Top award and, according to the commenter, that a national evaluation is a far more efficient method than using Race to the Top funds to pay for individual State-led evaluations. Another commenter emphasized the importance of a national evaluation of the Race to the Top program using State data. A few commenters recommended that we carry out both national and State-level evaluations of the Race to the Top program. Other commenters requested information on funding for Race to the Top evaluations, and two commenters recommended that up to 10 percent of Race to the Top awards be available to support those evaluations. One commenter expressed concern that the reporting requirements were focused on outcomes only, and did not include a description of the processes used to achieve those outcomes. Finally, four commenters suggested that a national evaluation should focus on identifying promising or best practices, while two commenters recommended the inclusion of ``process metrics'' to ensure that best practices can be fully documented to facilitate dissemination and adoption by others. Discussion: The Department appreciates this advice on how to structure an evaluation plan for the Race to the Top program. As described later in this notice, the Institute of Education Sciences (IES) will conduct a series of national evaluations of Race to the Top State grantees. The Department's goal for these evaluations is to ensure that its studies not only assess program impacts but also provide valuable information to State and local educators to help inform and improve their practices. We are not requiring through this notice that Race to the Top grantee States conduct independent evaluations. However, they are free to propose, within their applications, to use funds from Race to the Top to support independent evaluations. A full explanation of the Race to the Top evaluation plan is included in the Program Requirements section of this notice and the notice inviting applications. Changes: We have revised the Program Requirements section to reflect the evaluation requirements for all States that win a Race to the Top grant. Specifically, this notice has been revised to require State grantees to participate in a series of national evaluations that will be conducted by IES. This notice has been revised to reflect that these evaluations will involve components described further in this notice, including surveys, case studies, and evaluation of outcomes. We have further clarified that States have the option of conducting additional evaluations using Race to the Top funds or other funds. We have also revised this notice to reflect that State grantees, LEAs, and schools are expected to identify and share promising practices and make data available to help all States focus on continuous improvement. Participating LEA Scope of Work Comment: None. Discussion: The Program Requirement concerning Participating LEA Scope of Work is addressed in the discussion for Section A, State Success Factors. Change: The Program Requirement section is revised to include a requirement on Participating LEA Scope of Work. Making Work Available Comment: Two commenters suggested that the Department require that any new educational materials developed by Race to the Top State grantees be made available as open educational resources. One of these recommended that all outputs be open source and royalty-free. Several other commenters expressed concern about copyrighted intellectual property, proprietary systems, and the rights of contractors or partners, and that a requirement to share all outputs would preclude States from entering into contracts or licensing agreements or would conflict with agreements already in place. A commenter noted that one [[Page 59724]] specific State relies on subscriptions to copyrighted services for data warehousing and would have to build new systems to share data tools freely with the public. Two commenters suggested using the exclusion in the Statewide Longitudinal Data Systems grant program to protect intellectual property and proprietary products in Race to the Top. Discussion: We understand and agree with the concerns about proprietary information in the context of the proposed requirement that States and LEAs make available materials developed with Race to the Top funds. We are revising the Program Requirements section entitled Making Work Available to provide that such materials must be available ``unless otherwise protected by law or agreement as proprietary information.'' We also have clarified that this agreement applies to work developed under this grant. Changes: The Making Work Available requirement has been revised to read as follows: ``Unless otherwise protected by law or agreement as proprietary information, the State and its subgrantees must make any work (e.g., materials, tools, processes, systems) developed under its grant freely available to others, including but not limited to by posting the work on a Web site identified or sponsored by the Department.'' State Summative Assessments Comment: None. Discussion: The Program Requirement concerning State summative assessments is addressed in the discussion for Section B, Standards and Assessments. Changes: The Program Requirement Section is revised to include a program requirement on State summative assessments. Technical Assistance Comment: One commenter expressed support for the requirement that States participate in the Department's technical assistance activities. This commenter also suggested that technical assistance be provided by the federally supported research and development infrastructure, such as the regional labs. Another commenter argued that because successful implementation may be difficult, the Department should devote more resources and personnel to providing clear and fair technical assistance. One commenter recommended that the Department provide States with funds to cover the estimated costs of participating in technical assistance. Discussion: The Department intends to conduct extensive technical assistance activities related to Race to the Top grants and will utilize to the extent feasible all available resources, including federally supported research centers and regional laboratories, to support those activities. In addition, we will work to minimize the cost of this technical assistance to participants. Changes: None. Using Subgroups Under NAEP and the ESEA Comment: None. Discussion: The application requirement concerning use of subgroups under NAEP and the ESEA for reporting achievement gains and for setting future targets is addressed in the discussion for Section A, State Success Factors. Changes: We have added new paragraph (g) in the application requirements that explains the subgroup data that a State must provide in various parts of the application. A. State Success Factors Definitions: college enrollment, involved LEAs, participating LEAs. Comments regarding the preceding definitions are addressed, as appropriate, below. New Selection Criterion (A)(1)(i) Comment: None. Discussion: As noted elsewhere, we are adding a new section, ``State Success Factors,'' to the beginning of the Selection Criteria section in order to provide an opportunity for States to begin their Race to the Top proposals with a clear statement of their comprehensive and coherent statewide reform agendas. We are adding criterion (A)(1)(i) which will be used to assess the extent to which a State is successful in articulating the State's reform agenda. Changes: Criterion (A)(1) begins: ``Articulating the State's education reform agenda and LEAs' participation in it: The extent to which--(i) The State has set forth a comprehensive and coherent reform agenda that clearly articulates its goals for implementing reforms in the four education areas described in the ARRA and improving student outcomes statewide, establishes a clear and credible path to achieving these goals, and is consistent with the specific reform plans that the State has proposed throughout its application.'' Selection Criteria (A)(1)(ii) and (iii): Participating LEAs (proposed criteria (E)(3)(iv) and (E)(4)): Note: A number of comments common to criteria (A)(1)(iii) and (A)(3)(ii) are addressed in the discussion of (A)(3)(ii) later in this notice. Comment: Many commenters requested clarification regarding funding for LEAs under the Race to the Top program, State discretion to select participating LEAs, and whether LEAs may decline Race to the Top funding. Many commenters questioned whether State applications may exclude LEAs that are not committed to part or all of a State's Race to the Top plan. One commenter recommended giving States complete control over how Race to the Top funds are spent by participating LEAs, claiming that the State, not the LEA, will be held accountable for meeting Race to the Top goals and targets. Other commenters suggested that Race to the Top funds should be awarded only to LEAs that sign an agreement or otherwise fully agree to implement its State's Race to the Top plans. One commenter asked whether LEAs receiving a share of the 50 percent of Race to the Top funds distributed on the basis of the Title I, Part A formula under the ESEA are required to participate in the Race to the Top program. Several commenters asked if LEAs would be subject to Race to the Top requirements even if they declined to participate. Discussion: In response to these comments, and because LEAs are ultimately responsible for implementing many of the items in a State's Race to the Top plan, we have made a number of changes to provide great clarity on how LEAs can be involved in a State's plan. First, we are providing that LEAs can be included in States' Race to the Top projects at one of two levels: as ``participating LEAs'' or as ``involved LEAs.'' Participating LEAs, as defined in this notice, means LEAs that choose to work with the State to implement all or significant portions of the State's Race to the Top plan, as specified in each LEA's agreement with the State. Each participating LEA that receives funding under Title I, Part A will receive a share of the 50 percent of a State's grant award that the State must subgrant to LEAs, based on the LEA's relative share of Title I, Part A allocations in the most recent year, in accordance with section 14006(c) of the ARRA. Any participating LEA that does not receive funding under Title I, Part A (as well as one that does) may receive funding from the State's other 50 percent of the grant award, in accordance with the State's plan. States do not have the discretion to select participating LEAs; instead, each LEA will make the decision to sign on to the State's plan as a participating [[Page 59725]] LEA. All LEAs that agree to work with the State, and that sign valid agreements stating their commitment to implement all or significant portions of the State's plan (as defined by the State) must be included in the State's plan. States do have the flexibility to develop detailed reform plans in which LEAs must choose whether to participate. States also have the authority to define the ``significant portions'' of their Race to the Top plans that LEAs must agree to implement in order to qualify as participating LEAs. As described earlier, States that receive a Race to the Top grant must use at least 50 percent of the award to provide subgrants to their participating LEAs based on their relative shares of funding under Part A of Title I of the ESEA for the most recent year. Because all Race to the Top grants will be made in 2010, relative shares will be based on total funding received in FY 2009, including both the regular Title I, Part A appropriation and the amount made available by ARRA. The remaining funds will be available to the State for State-level activities and for disbursement to participating LEAs (regardless of their Title I eligibility), involved LEAs, or other entities, consistent with the State's plan. A State has no obligation to provide Race to the Top funds, benefits, or supports to non-participating LEAs. Participating LEAs must in turn use their funding in a manner that is consistent with the State's plan and its MOU or other binding agreement with the State. States may establish more detailed rules on uses of funds provided they are consistent with the ARRA, the terms of the grant award, and the Department's applicable administrative regulations. Although participating LEAs will receive subgrants from the State as described earlier, Race to the Top funds are not governed by the Title I restrictions on the uses of funds. As described earlier, participating LEAs agree to implement all or a significant portion of State's Race to the Top plans. However, other LEAs may choose to work with the State to implement those specific portions of the State's plan that require statewide or nearly statewide implementation, such as transitioning to a common set of K-12 standards. We have defined these LEAs in this notice as involved LEAs. As defined, involved LEAs do not receive a share of the 50 percent of a State's grant award that it must subgrant to LEAs in accordance with section 14006(c) of the ARRA, but States may provide other funding to involved LEAs under the State's Race to the Top grant in a manner that is consistent with the State's application. In general, involved LEAs are not included in, and are not subject to, the requirements of a State's Race to the Top plan. It is important to note that this notice does not require LEAs to participate in a State's plan (whether as participating or as involved LEAs) or give States the authority to impose such a requirement. Rather, through the definitions of participating LEA and involved LEA, we are setting the parameters for what LEAs must do to be eligible for certain funding streams. In addition, through absolute priority 1, the Department is specifying that States will only be awarded grants if they demonstrate sufficient LEA participation and commitment to successfully implement and achieve the goals of their plans; and through criteria (A)(1)(ii) and (A)(1)(iii), this notice sets forth the terms by which reviewers will award points to each State based on the participation and commitment of their LEAs. Changes: We have added two new definitions to this notice. The definition of participating LEAs clarifies that participating LEAs choose to work with the State to implement all or significant portions of the State's Race to the Top plan, as specified in each LEA's agreement with the State. Each participating LEA that receives funding under Title I, Part A will receive a share of the 50 percent of a State's grant award that the State must subgrant to LEAs, based on the LEA's relative share of Title I, Part A allocations in the most recent year, in accordance with section 14006(c) of the ARRA. Any participating LEA that does not receive funding under Title I, Part A (as well as one that does) may receive funding from the State's other 50 percent of the grant award, in accordance with the State's plan. The definition of involved LEAs clarifies that such LEAs choose to work with the State to implement those specific portions of the State's plan that necessitate full or nearly-full statewide implementation, such as transitioning to a common set of K-12 standards (as defined in this notice). Involved LEAs do not receive a share of the 50 percent of a State's grant award that it must subgrant to LEAs in accordance with section 14006(c) of the ARRA, but States may provide other funding to involved LEAs under the State's Race to the Top grant in a manner that is consistent with the State's application. Comment: Many commenters recommended that the Department define ``participating school'' in the final notice. Discussion: Participating LEAs are responsible for determining the roles and responsibilities of their schools in Race to the Top activities; these should be consistent with the LEA's agreement with the State. Consequently, we do not believe that there is a need for a definition of participating school in this notice. Changes: None. Comment: Commenters requested additional clarification pertaining to how States would identify and account for LEA participation and support in State reform plans. Multiple commenters recommended that participating LEAs and charter schools formally declare their support in writing as part of the Race to the Top application. One commenter recommended requiring States to list all the LEAs that requested to be included in designing and developing the State plan. Discussion: Proposed criterion (E)(3)(iv) was included to elicit information about the extent of the commitment to and participation of LEAs in a State's Race to the Top plan. Because we believe that States should begin their Race to the Top proposals with clear statements of their entire reform agendas, and because LEA implementation is a central component of that agenda, we are moving this criterion into the new ``State Success Factors'' section. Furthermore, to add clarity, we are dividing the proposed criterion into two revised criteria. In this final notice, criterion (A)(1)(ii) addresses the level of commitment among participating LEAs, while criterion (A)(1)(iii) addresses the extent of LEA participation. Because the extent of LEA participation should be measured partly by the expected effects on student outcomes statewide, we have incorporated into criterion (A)(1)(iii) the language from proposed criterion (E)(4) regarding a State's goals for increasing student achievement, decreasing achievement gaps, and increasing graduation rates. As discussed later, we also include new criterion (A)(1)(iii)(d) regarding increasing college enrollment and credit accumulation. In addition, as evidence to support the State's response to criteria (A)(1)(ii) and (A)(1)(iii), Appendix A to this notice asks States for the following information: (1) An example of the State's standard participating LEA MOU and description of variations used, if any; (2) the completed summary table indicating which specific portions of the State's plan each LEA is committed to implementing and relevant summary statistics; (3) the completed summary table, indicating which LEA leadership signatures have been obtained; (4) the completed summary table, indicating [[Page 59726]] the numbers and percentages of participating LEAs, schools, K-12 students, and students in poverty; (5) tables and graphs that show the State's goals, overall and by subgroup, requested in criterion (A)(1)(iii), together with the supporting narrative; and (6) the completed detailed table, by LEA, that includes the information requested in criteria (A)(1)(ii) and (A)(1)(iii). As discussed in greater detail elsewhere in this notice, the Department is providing a sample MOU (see Appendix D) to assist States and LEAs during this process. Changes: Criterion (A)(1)(ii) specifies that reviewers will evaluate the extent to which the participating LEAs are strongly committed to the State's plans and to effective implementation of the four education reform areas, as evidenced by Memoranda of Understanding (MOUs) (as set forth in Appendix D) or other binding agreements between the State and its participating LEAs (as defined in this notice) that include-- (a) Terms and conditions that reflect strong commitment by the participating LEAs (as defined in this notice) to the State's plans; (b) Scope-of-work descriptions that require participating LEAs (as defined in this notice) to implement all or significant portions of the State's Race to the Top plans; and (c) Signatures from as many as possible of the LEA superintendent (or equivalent), the president of the local school board (or equivalent, if applicable), and the local teachers' union leader (if applicable) (one signature of which must be from an authorized LEA representative) demonstrating the extent of leadership support within participating LEAs (as defined in this notice). In addition, criterion (A)(1)(iii) specifies that LEA participation will be evaluated based on the extent to which the LEAs that are participating in the State's Race to the Top plans (including considerations of the numbers and percentages of participating LEAs, schools, K-12 students, and students in poverty) will translate into broad statewide impact, allowing the State to reach its ambitious yet achievable goals, overall and by student subgroup, for-- (a) Increasing student achievement in (at a minimum) reading/ language arts and mathematics, as reported by the NAEP and the assessments required under the ESEA; (b) Decreasing achievement gaps between subgroups in reading/ language arts and mathematics, as reported by the NAEP and the assessments required under the ESEA; (c) Increasing high school graduation rates (as defined in this notice); and (d) Increasing college enrollment (as defined in this notice) and increasing the number of students who complete at least a year's worth of college credit that is applicable to a degree within two years of enrollment in an institution of higher education. Finally, Appendix A, Evidence and Performance Measures, has been revised to specify the evidence that States must submit when responding to criteria (A)(1)(ii) and (A)(1)(iii). Memoranda of Understanding (MOUs) Comment: Many commenters requested clarification regarding the MOUs between States and participating LEAs, including the purpose, requirements, and expected contents of the MOUs. Discussion: We agree with the commenters that additional clarification is needed on the purpose and content of the MOUs. As discussed earlier, we are clarifying in criterion (A)(1)(ii) the elements of the MOU or other binding agreements that reviewers will consider in evaluating LEA commitment. We also are adding a new requirement that clarifies the expectations for the Participating LEA scope of work. Finally, we are including in Appendix D to this final notice a model MOU to provide further guidance to States in preparing these agreements with their LEAs. Changes: We have added to the program requirements a new Participating LEA Scope of Work requirement, which clarifies that the agreements signed by participating LEAs (as defined in this notice) must include a scope-of-work section. The scope of work submitted by LEAs and States as part of their Race to the Top applications will be preliminary. Preliminary scopes of work should include the portions of the State's proposed reform plans that the LEA is agreeing to implement. If a State is awarded a Race to the Top grant, its participating LEAs (as defined in this notice) will have up to 90 days to complete final scopes of work, which must contain detailed work plans that are consistent with their preliminary scopes of work and with the State's grant application, and should include the participating LEAs' specific goals, activities, timelines, budgets, key personnel, and annual targets for key performance measures. We have added a new Appendix D to this notice which provides a model MOU that States may use in developing these agreements. Comment: One commenter suggested that final agreements with participating LEAs should be based on the actual amount of funding a State receives and, therefore, that States should not be required to provide detailed MOUs with their applications. Discussion: The Department agrees that LEAs should not have to provide final agreements detailing their precise activities at the time that States apply, and as discussed earlier, we are clarifying in the new Participating LEA Scope of Work requirement that States will have 90 days after the receipt of a grant to negotiate the final scope of work agreements with their participating LEAs. However, we believe that it is critical that LEAs indicate, at the time they sign their MOU in connection with the State's application, which parts of the State's plan they will participate in implementing. Peer reviewers must have this information in order to determine, under criterion (A)(1)(ii), whether the State's participating LEAs are indeed strongly committed to the State's plan. We also note that, because we are providing nonbinding budget ranges in the notice inviting applications and encouraging States to propose budgets that match the plans they propose, States should have some sense of the expected funding available for LEAs before they apply for their grants. Changes: None. Comment: One commenter recommended that the Department accept a signed ``certification of consultation,'' rather than an MOU. The commenter stated that such a certification would be the more appropriate method for demonstrating agreement in the commenter's State. Discussion: We understand that States may have processes and procedures other than an MOU that they use to establish agreements with their LEAs. As long as such certifications or agreements are binding, they may be included in a State's application as evidence of its LEAs' commitment to its reform plan. We are adding language in criterion (A)(1)(ii) to make this clear. Changes: Criterion (A)(1)(ii) provides that participating LEAs' commitment to the State's plans may be evidenced by an MOU or other binding agreement. Comment: A few commenters stated that it would be burdensome and time-consuming to require MOUs between an SEA and its LEAs with required signatories, and suggested that the Department allow SEAs to design and propose a stakeholder input process in accordance with State and local needs. One commenter requested clarification [[Page 59727]] as to whether a State's Race to the Top application must include an MOU with each LEA or whether an outline of what would be covered in an MOU with an LEA would suffice. Discussion: The Department acknowledges that requiring States to develop and obtain signed MOUs for submission with Race to the Top applications on a short timeline will be a challenge. However, strong LEA participation in State Race to the Top plans is essential if those plans are to have a broad impact on student outcomes. To assist States in this work, we are providing, as part of the application package and Appendix D in this notice, a model MOU that States can adapt or use in signing agreements with their participating LEAs. With regard to the question of whether a State's Race to the Top application must include an MOU with each LEA or whether an outline of what would be covered in an MOU with an LEA would suffice, criterion (A)(1)(ii) makes clear that the MOUs included in a State's application will be used as evidence of LEAs' commitment to the State's plan. Therefore, in order to receive maximum points on criterion (A)(1)(ii), a State should have an MOU for each participating LEA. However, in acknowledgement of the short timeline, we are clarifying in the new Participating LEA Scope of Work requirement that a State need only include preliminary scopes of work from its participating LEA in its application. States will have up to 90 days after receiving a grant award to obtain the final scope of work from participating LEAs. States also can use this time to reach agreements with additional participating LEAs. Changes: We have included in Appendix D to this notice a model MOU that States can adapt or use for their LEAs who will be participating LEAs. In addition, we have added a new Participating LEA Scope of Work requirement in order to clarify that the MOUs need only include a preliminary scope of work, which must be finalized within 90 days of the State receiving a Race to the Top award. This requirement also clarifies that winning States can reach agreements with additional participating LEAs within 90 days of the State receiving a Race to the Top award. Comment: Several commenters requested that the MOU between the State and its LEAs require the signature of the president of the local PTA units and State charter school membership associations. Another commenter requested that State union leaders be required to approve the State's entire application. Discussion: The Department agrees that Race to the Top plans would benefit from input and involvement by parents, teachers, and the organizations that represent them. Thus, at the State level, criterion (A)(2)(ii) includes teachers' unions, parent-teacher organizations, and charter school membership associations among the broad group of stakeholders from which a State could obtain statements or actions of support to demonstrate statewide commitment to its Race to the Top plan. In addition, at the LEA level, criterion (A)(1)(ii)(c) specifies that LEA leadership support will be evaluated based on the number of signatures gathered from among the superintendent (or equivalent), school board president (or equivalent, if applicable), and teachers' union leader (if applicable). Changes: None. Comment: Two commenters recommended removing the phrase ``ambitious yet achievable'' in proposed criterion (E)(4) on the grounds that it might encourage States to set a low bar and that it reflects a step backward from current ESEA accountability requirements emphasizing 100 percent proficiency for all students. A number of commenters requested that the Department provide more guidance on expectations for State targets. Discussion: We are retaining the ``ambitious yet achievable'' language in criterion (A)(1)(iii) (proposed criterion (E)(4)). The Department believes that this language strikes the right balance between encouraging States to set a high bar for Race to the Top goals while recognizing that real change in education is difficult and takes time. The purpose of this language is to encourage realistic thinking and planning that connects specific activities to specific achievable results. Further, the Department believes that the competitive aspect of the Race to the Top program will prevent States from setting low bars. Changes: None. Comment: One commenter objected to our proposal that in responding to proposed criterion (E)(4), regarding targets for improved student outcomes, States submit an estimate of the State's expected levels of future performance were the State not to receive Race to the Top funding; this commenter argued that a State's goal should be the same with or without additional funding. Another commenter requested clarification as to how such outcomes should be estimated. Discussion: Because this requested piece of evidence was confusing to States, we have decided not to include it in the final notice. Changes: The final notice does not ask States to provide estimates of their expected levels of future performance were they not to receive funds under this program. Selection Criterion (A)(2)(ii): Stakeholder Support (proposed criterion (E)(3)): Comment: We received many comments on the list of stakeholders in proposed criterion (E)(3) from which States could enlist support and commitment for their State plans. Many commenters welcomed the broad list of stakeholders; in particular, several commenters expressed appreciation for including teachers' unions in the list of stakeholders given the need for teacher and school staff support to effectively implement Race to the Top reforms. A few commenters recommended adding principals to the list of stakeholders. Some commenters recommended that States obtain the signature of union leaders on their applications, while another recommended that teachers' unions not be given ``veto power'' over statewide or local plans. Discussion: We agree with commenters that it is important for States to obtain support for their reform plans from teachers and principals, and that this should include a State's teachers' union or statewide teachers' association. As stewards of the teaching workforce, teachers' unions have a critical role to play in education reform. Therefore, in this final notice, criterion (A)(2)(ii)(a) (proposed criterion (E)(3)) specifically identifies teachers and principals, which include a State's teachers' union or statewide teachers' association, as stakeholders whose support will earn States points. However, we decline to require States to obtain signatures from union leaders in order to apply for a Race to the Top Grant. Note that for clarity, we have moved ``charter school authorizers'' from this list to the list in criterion (A)(2)(ii)(b), regarding other critical stakeholders. Changes: Criterion (A)(2)(ii)(a) provides for evaluation of a State's application based on the extent to which it has a high-quality plan to use the support from its teachers and principals, which include the State's teachers' unions or statewide teacher associations, to better implement its plans. Comment: Some commenters stated that State plans should not include elements that potentially undermine collective bargaining agreements. Discussion: We agree with the comment that State reform plans should [[Page 59728]] not undermine collective bargaining agreements. We also believe that Race to the Top may lead to forward-thinking approaches that change how LEAs and teachers' unions work together within the framework of collective bargaining. Of course, any changes to collective bargaining agreements must be collectively bargained. Changes: None. Comment: Many commenters recommended that other stakeholder groups be included in proposed criterion (E)(3) as groups from which States should obtain support and commitment for their State plans. Commenters recommended that the following groups be included: State legislatures, charter school associations, parent and family organizations, parent- teacher associations, Parent Information and Resource Centers, youth- serving community-based organizations (CBOs) and other community groups, CBOs serving Native American tribes, higher education leaders and providers, members of the business community, private and faith- based school leaders, students, local education funds, value-added intermediaries, public broadcasting entities, municipal leaders, teachers and principals who have successfully turned around schools, school service providers, guidance counselors, statewide after-school networks, and statewide teacher associations. Discussion: We appreciate the broad and diverse group of stakeholders that commenters identified as important to States' reform efforts. Obviously, the stakeholders from which a State should garner support for its reform plan will vary based, to a large extent, on the unique needs of the State and its LEAs. While we cannot include all of the stakeholders recommended by commenters in this notice, we believe it is important to include several examples for illustrative purposes and to encourage States, as appropriate to their unique contexts, to solicit broad support. We are, therefore, designating proposed criterion (E)(3)(ii) as criterion (A)(2)(ii)(b), and adding ``charter school authorizers'' from proposed criterion (E)(3)(i), as well as additional stakeholders from whom the State may want to obtain support for its plans. Changes: Criterion (A)(2)(ii)(b) reads as follows: ``Other critical stakeholders, such as the State's legislative leadership; charter school authorizers and State charter school membership associations (if applicable); other State and local leaders (e.g., business, community, civil rights, and education association leaders); Tribal schools; parent, student, and community organizations (e.g., parent-teacher associations, nonprofit organizations, local education foundations, and community-based organizations); and institutions of higher education.'' Comment: Some commenters viewed proposed criterion (E)(3) as an opportunity to be involved in developing a State's reform plan. One commenter recommended adding language to the final notice to require LEA participation in the development of the State plan, while another commenter proposed that States develop their plans in consultation with civil rights leaders, parents, and community groups that are representative of the State's population, and document such consultation. Other commenters recommended that the Department award additional points for State plans that coordinate and integrate support from education, health, nutrition, social services, and juvenile justice stakeholders, or for demonstrating a broad spectrum of stakeholder support. Discussion: There is no requirement that a State involve its LEAs, or any other persons or groups, in developing its reform plan. However, given that the success of a State's plan depends, to a large extent, on the support and commitment of its LEAs to implement the plan, we strongly encourage States to work together with their LEAs in developing their State plan. Similarly, we believe that committed and interested stakeholders can make the difference in a reform's success or failure. We decline to require States to develop their plans with any specific stakeholders or to award additional points for plans that coordinate with specific groups or agencies, as recommended by commenters. We believe the decision on who to work with in developing a State plan is best left to States. Changes: None. Comment: A number of commenters expressed concern that requiring support or input from a broad range of stakeholders could lead to less rigorous, ``watered-down'' plans if States were to satisfy all the different groups with their competing interests. Some of these commenters recommended eliminating the provision on stakeholder support from the final notice, while others suggested clarifying that ``buy- in'' from all stakeholders is not required. Several commenters requested a definition of ``statewide support.'' Discussion: Race to the Top does not require States to work with specific stakeholders (other than LEAs) or obtain their support and commitment in order to be eligible for a grant. Instead, States will earn points for demonstrating stakeholder support under criterion (A)(2)(ii). In addition, we note that the list of proposed stakeholders in criterion (A)(2)(ii)(b) is illustrative. We believe that this list provides sufficient clarity regarding the phrase ``statewide support'' and, therefore, decline to define it in this notice. Changes: None. Comment: Some commenters requested that the Department include in the final notice examples of the specific kinds of evidence that should be used to demonstrate stakeholder support. For example, one commenter suggested that evidence of support should include strong letters of commitment from teachers' unions; another commenter suggested that States provide documentation that plans were developed with stakeholder support. Discussion: We agree that it would be helpful to specify the evidence that a State should submit to demonstrate the strength of its support from a broad range of stakeholders. To give further guidance as to how States should respond to this criterion, we are revising criterion (A)(2)(ii) to clarify that reviewers will judge the extent to which a State has a high-quality plan to use its stakeholder support to better implement its Race to the Top plans, as evidenced by the strength of its stakeholders' statements or actions of support. We are also clarifying in Appendix A to this notice that States should provide the key statements or actions of support and a summary of them in their applications. Changes: We have added to the introduction in criterion (A)(2)(ii), the following: ``Use support from a broad group of stakeholders to better implement its plans, as evidenced by the strength of the statements or actions of support from--.'' We have changed the requested evidence in Appendix A to require that States provide ``a summary in the narrative of the statements or actions and inclusion of key statements or actions in the Appendix'' when responding to this criterion. Selection Criterion (A)(2): Building State Capacity (proposed criterion (E)(5)): Comment: A number of commenters expressed support for criterion (A)(2) (proposed criterion (E)(5)), which focuses on a State's plan to build statewide capacity to implement, scale up, and sustain its reform plan. One commenter in particular emphasized the importance of plan implementation. This commenter claimed that States [[Page 59729]] often make empty promises and fail to deliver on their grant applications. Discussion: We agree that the Race to the Top competition must judge States' capabilities to implement their plans, as well as the quality of the plans themselves. To emphasize this point, we are moving most of the criteria in proposed criterion (E)(5) to criterion (A)(2)(i), in which the Department will evaluate the extent to which a State has a high-quality plan to ensure it has the capacity necessary to implement its proposed Race to the Top plans. We are adding a criterion regarding State leadership. We are also including in criterion (A)(2)(i)(c) (proposed (E)(5)(i)) more specific examples of activities that support effective and efficient grant administration, such as budget reporting and monitoring, performance measure tracking and reporting, and fund disbursement. Changes: Criterion (A)(2)(i)(a) has been added to address the extent to which a State has a high-quality plan to provide strong leadership and dedicated teams to implement the statewide education reforms plans the State has proposed. Criterion (A)(2)(i)(c) incorporates with minor changes the language from proposed criterion (E)(5)(i) and now reads: ``Providing effective and efficient operations and processes for implementing its Race to the Top grant in such areas as grant administration and oversight, budget reporting and monitoring, performance measure tracking and reporting, and fund disbursement.'' Comment: Some commenters supported proposed criterion (E)(5)(ii) and its focus on ensuring the dissemination of best practices. Discussion: We agree that supporting LEAs to implement the State's reform plans and disseminate successful practices is critical to a State's reform efforts. Therefore, we are re-designating proposed criterion (E)(5)(ii) as criterion (A)(2)(i)(b) and adding examples of State activities that will help LEAs successfully implement reform plans, such as identifying promising practices, evaluating the effectiveness of these practices, ceasing ineffective practices, and widely disseminating and replicating effective practices. Changes: We have re-designated proposed criterion (E)(5)(ii) as criterion (A)(2)(i)(b) and added additional text for clarity and completeness. Criterion (A)(2)(i)(b) now reads as follows: ``Supporting participating LEAs (as defined in this notice) in successfully implementing the education reform plans the State has proposed, through such activities as identifying promising practices, evaluating these practices' effectiveness, ceasing ineffective practices, widely disseminating and replicating the effective practices statewide, holding participating LEAs (as defined in this notice) accountable for progress and performance, and intervening where necessary.'' Comment: Some commenters suggested that the Department require coordination between State agencies and education-related organizations, for example, to share and scale up the adoption of successful Race to the Top strategies. Other commenters requested clarification regarding the collaboration contemplated by the Department in proposed criterion (E)(5)(iv), which would examine the quality of a State's plan to collaborate with other States on key elements of a State's application. Another commenter suggested that the Department strengthen this collaboration requirement. Discussion: We agree that States and LEAs should partner with and learn from outside organizations, other agencies, and other States and LEAs whenever doing so would help them improve student outcomes. However, commenters' confusion over the Department's intentions around collaboration convinced us that reviewers would be best able to reliably score State applications if collaboration were evaluated in the context of specific plans rather than as a stand-alone portion of a State's application. In other words, to the extent that a State improves the quality of its plan in response to a given criterion by collaborating with others, the State will receive credit under that criterion for having a high-quality plan. In addition, in situations where there is especially clear value to collaboration among States, such as in the development of common standards and assessments (see criteria section B), we have specifically encouraged collaboration. We have therefore removed from this notice the more general criterion on collaboration (proposed criterion (E)(5)(iv)). Changes: We have removed proposed criterion (E)(5)(iv), regarding collaboration with other States, from this final notice. Comment: Some commenters emphasized the need for States to ensure that LEAs have sufficient resources to implement reforms. Discussion: We agree with the commenters that LEA activities are central to Race to the Top and that LEAs will need sufficient resources to make their activities a success. In the NPP, proposed application requirement (e) required a State to include a budget that detailed, among other things, how it would use grant funds and other resources to meet targets and perform related functions. In this notice, we have retained that application requirement (re-designated as application requirement (c)), but also included language in criterion (A)(2)(i)(d) directing reviewers to evaluate how the State will use its Race to the Top funds to accomplish its plans and meet its targets. We also note that, under section 14006(c) of the ARRA, States must subgrant at least 50 percent of their Race to the Top grant to participating LEAs based on LEAs' relative shares of funding under Part A, Title I of the ESEA. In addition, States have considerable flexibility in awarding or allocating the remaining 50 percent of their Race to the Top awards, which are available for State-level activities, disbursements to LEAs, and other purposes as the State may propose in its plan. Changes: Criterion (A)(2)(i)(d) provides for the evaluation of the extent to which the State has a high-quality plan for using the funds for this grant, as described in the State's budget and accompanying budget narrative, to accomplish the State's plans and meet its targets, including, where feasible, by coordinating, reallocating, or repurposing education funds from other Federal, State, and local sources so that they align with the State's Race to the Top goals. Comment: A number of commenters expressed concern regarding proposed criterion (E)(5)(v), which focuses on the extent to which States coordinate, allocate, or repurpose funds from other sources to align with the State's Race to the Top goals. One commenter suggested that it was beyond the scope of the Race to the Top program to suggest that non-ARRA funds be reallocated to meet the goals of the Race to the Top program. A number of commenters requested that the Department add the phrase ``consistent with program requirements'' after proposed criterion (E)(5)(v) to ensure that reallocation of funds does not violate the program requirements of the IDEA. Discussion: In response to concerns raised by many commenters regarding a State's ability or authority to repurpose education funds from other sources to align with a State's Race to the Top plan, we are adding ``where feasible'' in proposed criterion (E)(5)(v). We also are re-designating proposed criterion (E)(5)(v) as criterion (A)(2)(d) and adding additional text for clarity and completeness. However, we continue to believe that States need to focus and align their education funding resources for maximum impact consistent with [[Page 59730]] existing program requirements, and that Race to the Top should encourage States to leverage the improved use of all available resources, regardless of the source, to support effective, comprehensive changes in State and local education systems. In this context, consideration of the extent to which a State is willing to realign available resources in support of Race to the Top goals is not only appropriate, but necessary. Changes: We have re-designated criterion (E)(5)(v) as criterion (A)(2)(d) and clarified that States will be judged based on their coordination, reallocation, or repurposing of education funds so that they support Race to the Top goals ``where feasible.'' Comment: One commenter recommended amending proposed criterion (E)(5)(iii) to include fiscal resources, rather than ``economic resources'' in the list of resources that States should use to continue Race to the Top reforms after the grant funding. Another commenter recommended clarifying that grant activities should be continued only if there is evidence of success. Discussion: We agree that ``fiscal'' is a better word than ``economic'' to describe the financial resources that a State will use to continue Race to the Top reforms after the period of Race to the Top funding has ended. Therefore, we are changing proposed criterion (E)(5)(iii) to refer to fiscal resources and re-designating criterion (E)(5)(iii) as criterion (A)(2)(i)(e). In addition, we are adding language to criterion (A)(2)(i)(e) to clarify that post-Race to the Top grant planning applies only to continuing support for Race to the Top activities for which there is evidence of success. Changes: We have re-designated proposed criterion (E)(5)(iii) as criterion (A)(2)(i)(e) and revised the criterion to read as follows: ``Using the fiscal, political, and human capital resources of the State to continue, after the period of funding has ended, those reforms funded under the grant for which there is evidence of success.'' Selection Criterion (A)(3): Demonstrating Significant Progress in Raising Achievement and Closing Gaps (proposed criteria (E)(1) and (E)(4)): Note: This section includes issues common to criteria (A)(1)(iii) and (A)(3)(ii). Comment: None. Discussion: The ARRA emphasizes the importance of States demonstrating significant progress in meeting the objectives of the four assurance areas. In the NPP, proposed criterion (E)(1)(i) asked States to describe their progress in each of the four education reform areas generally, proposed criterion (E)(1)(ii) asked States to describe how they have used ARRA and other Federal and State funding to pursue reforms in these areas, and proposed criterion (E)(1)(iv) asked States to describe the successes they have had in increasing student achievement, closing achievement gaps, and increasing graduation rates. In order to reduce redundancy and the burden on States, we are combining proposed criteria (E)(1)(i) and (E)(1)(ii) into one criterion and designating it as criterion (A)(3)(i). We are also designating proposed criterion (E)(1)(iv) as criterion (A)(3)(ii). Both of these revised criteria are now part of the State Success Factors section. We believe this reorganization more logically groups our requests for information regarding progress. We have also added, in criterion (A)(3)(ii), that States may report progress since ``at least'' 2003 to allow a longer data history for States that have such data (all States have NAEP and ESEA data since 2003, but not all States participated in all of NAEP prior to 2003). Further changes to criterion (A)(3)(ii) are discussed later in this section. Changes: We have combined proposed criteria (E)(1)(i) and (E)(1)(ii) into one criterion, designated (A)(3)(i), and designated proposed criterion (E)(1)(iv) as criterion (A)(3)(ii). Criterion (A)(3) now evaluates a State based on the extent to which the State has demonstrated its ability to-- (i) Make progress over the past several years in each of the four education reform areas, and used its ARRA and other Federal and State funding to pursue such reforms; (ii) Improve student outcomes overall and by student subgroup since at least 2003, and explain the connections between the data and the actions that have contributed to-- (a) Increasing student achievement in reading/language arts and mathematics, both on the NAEP and on the assessments required under the ESEA; (b) Decreasing achievement gaps between subgroups in reading/ language arts and mathematics, both on the NAEP and on the assessments required under the ESEA; and (c) Increasing high school graduation rates. Comment: A number of commenters objected to our proposal that States demonstrate progress in increasing student achievement and closing the achievement gap using the National Assessment of Educational Progress (NAEP). Some of these commenters asserted that the NAEP provides an incomplete and distorted view of student achievement, particularly the achievement of students with disabilities. Another commenter noted that the NAEP does not include high school results. Others expressed concern that using the NAEP data would only encourage teaching to a test or would conflict with the NAEP's purpose as an outside and valid measurement. Several commenters stated that, in addition to the NAEP, the Department should allow States to demonstrate achievement gains on assessments or achievement measures under the ESEA, such as the annual proficiency scores and targets used to determine adequate yearly progress (AYP), including proficiency rates broken down by subgroup. One commenter stated that it would be particularly unfair to require a State to use NAEP data where the State could demonstrate that it has more rigorous assessments. Other commenters suggested the final notice permit States to include other measures to demonstrate achievement gains. Discussion: The Department proposed using NAEP results to measure State progress in increasing student achievement and decreasing achievement gaps because NAEP is the only national measure of student achievement that is comparable across States. The limitations of the NAEP, as pointed out by commenters, are well-known: It is not aligned to State content standards, does not include high school results, and may not provide accurate achievement information for students with disabilities and certain other subgroups. Also, the NAEP is not administered annually, limiting the number of data points available for measuring progress toward Race to the Top goals. However, the ability of NAEP to compare progress across States and to be a consistent measure over time remains a compelling reason to use it for Race to the Top. Accordingly, we believe that including data from both the NAEP and the annual State assessments required under the ESEA will provide a more complete and valid picture of State progress to date and States' goals for increasing student achievement and decreasing achievement gaps. We are incorporating with some revisions the language from proposed criteria (E)(1)(iv) and (E)(4) into criteria (A)(3)(ii) and (A)(1)(iii) to reflect this decision. In addition, we are specifying in application requirement (g) that when describing data for the assessments required under the ESEA, the State should note any factors (e.g., changes in cut scores) that would impact the comparability of data from one year to the next. We also note that [[Page 59731]] including more than one assessment should significantly reduce any risks of teaching to the test. As a result, we do not believe that including this use of the NAEP in Race to the Top will affect NAEP's validity or utility as an objective measure of student achievement, as suggested by commenters. Regarding the comment that we should allow States to demonstrate achievement gains on assessments or achievement measures under the ESEA, such as the annual proficiency scores and targets used to determine AYP, we note that States already issue annual reports on AYP status for schools and LEAs, including proficiency rates for all schools; there is no need to duplicate this reporting by requiring its inclusion in a State's annual Race to the Top report. However, States that desire to include AYP data (or other measures) in their annual Race to the Top reports would be free to do so. Changes: Proposed criteria (E)(1)(iv) and (E)(4) have been redesignated as criteria (A)(3)(ii) and (A)(1)(iii), respectively. They have been revised to consider both NAEP and ESEA assessment results when evaluating increases in student achievement and decreases in achievement gaps in reading/language arts and mathematics; criterion (A)(3)(ii) considers these in terms of historic gains (since at least 2003), while criterion (A)(1)(iii) considers them in terms of future goals in light of the participation of the State's LEAs in the State's reform plans. The evidence requested in Appendix A has also been revised to conform with the criteria. We have also added application requirement (g), which we discuss in more detail later in this notice. Comment: Many commenters recommended modifications or additions to the achievement measures for assessing past progress and setting future targets in proposed criteria (E)(1)(iv) and (E)(4). Other commenters supported the NPP's emphasis on increasing student achievement, narrowing achievement gaps, and increasing graduation rates. One key area of concern for several commenters was dropout recovery and prevention, with one commenter recommending that the Department supplement existing measures on graduation rates in proposed criteria (E)(1)(iv) and (E)(4)(iii) with targets for decreasing the number of young people aged 18 to 24 without a high school diploma. Other commenters recommended that States set targets and report on the percentage of low-income and minority 9th grade students who graduate from high school in four years, the number of low-income and minority students who are on track to be college- and career-ready, and increases in the percentage of low-income and minority students being taught by effective teachers. Other commenters recommended the addition of targets for early childhood education, such as goals for kindergarten readiness and third-grade reading and mathematics. A few commenters suggested that in evaluating Race to the Top applications, the Department consider the extent to which a State has ambitious annual targets for increasing college enrollment and completion rates or increasing college and career readiness. Discussion: The Department acknowledges that many measures could demonstrate progress toward Race to the Top goals. We especially agree that increasing college enrollment is an important area that should be reviewed in the context of Race to the Top. We are, therefore, adding criterion (A)(1)(iii)(d), which examines the extent to which a State's LEA participation will allow the State to reach its ambitious yet achievable goals for increasing college enrollment and credit accumulation. We are also adding a definition of college enrollment to help States respond appropriately to this criterion. After careful consideration of the comments, the Department believes that this new criterion, in combination with the proposed measures--which focus on reading, mathematics, and increasing graduation rates--reflect the right emphasis on key areas that States can report on with some validity and comparability. Further increasing the number of measures would increase data collection and reporting burdens on States and LEAs, many of which have not been collecting data in the areas suggested by commenters. States that want to include their own supplemental measures and targets are free to do so, and the ongoing expansion of State data systems, which is supported by the Race to the Top program and encouraged under invitational priority 4, will likely facilitate future indicators and targets in such areas as early childhood, drop-out prevention, and student mobility. Changes: We have added criterion (A)(1)(iii)(d), which rewards States whose LEA participation will translate into broad statewide impact, allowing the State to reach its ambitious yet achievable goals, overall and by student subgroup, for increasing college enrollment (as defined in this notice) and increasing the number of students who complete at least a year's worth of college credit that is applicable to a degree within two years of enrollment in an institution of higher education. We have also added a definition of college enrollment, which refers to the enrollment of students who graduate from high school consistent with 34 CFR 200.19(b)(1) and who enroll in an institution of higher education (as defined in section 101 of the Higher Education Act, Public Law 105-244, 20 U.S.C. 1001) within 16 months of graduation. Comment: Many commenters requested that the Department ensure that State applicants set targets for all core academic subjects reported by the NAEP, and not only in reading and mathematics, as in proposed criteria (E)(4)(i) and (ii). Discussion: The final notice continues to focus on reading and mathematics achievement, partly to ensure consistency with ESEA assessment requirements and partly to promote comparability, since all States have NAEP and ESEA assessment results dating back to at least 2003 in those subjects. The Department notes, however, that these are minimum expectations; States may include assessment results in other subjects both to demonstrate past progress and to measure Race to the Top performance going forward. Changes: None. Comment: Many commenters recommended that States focus more narrowly on specific student groups in crafting their State Plans to raise student achievement and close achievement gaps, including among high-need students. Discussion: We agree with the commenters that closing achievement gaps is an urgent national priority. Proposed criterion (E)(4) asked States to set ambitious yet achievable goals for closing achievement gaps, as well as for increasing student achievement and graduation rates overall and by subgroup. Criterion (A)(1)(iii) in this final notice retains these provisions and includes similar subgroup-specific goals in new criterion (A)(1)(iii)(d), regarding college enrollment and credit accumulation. This final notice also includes new language in criterion (A)(3)(ii) specifying that States' recent gains in increasing student achievement and graduation rates will be evaluated both overall and by student subgroup. We leave it to States to determine which of the subgroups in their student populations need the most attention. Changes: Criterion (A)(3)(ii) rewards States that have demonstrated the ability to improve student outcomes overall and by student subgroup since at least 2003 and explain the connections [[Page 59732]] between the data and the actions that have contributed to-- (a) Increasing student achievement in reading/language arts and mathematics, both on the NAEP and on the assessments required under the ESEA; (b) Decreasing achievement gaps between subgroups in reading/ language arts and mathematics, both on the NAEP and on the assessments required under the ESEA; and (c) Increasing high school graduation rates. Comment: A number of commenters suggested that the Department should not ask States to report data disaggregated by the student subgroups in section 303(b)(2)(G) of the NAEP but rather use the student subgroups as described in section 1111(b)(2)(C)(v)(II) of the ESEA. Others emphasized the importance of disaggregating data by subgroup, including race and gender. Discussion: We agree with the need to clarify the subgroups for which States must report achievement data given the differences in reporting achievement data by subgroups under the NAEP versus under the ESEA. As discussed earlier, we are adding new paragraph (g) in the application requirements that explains the subgroup data that a State must provide in various parts of the application. Specifically, when addressing items in the criteria for student subgroups with respect to the NAEP, the State must provide data using the NAEP subgroups as described in section 303(b)(2)(G) of the National Assessment of Educational Progress Authorization Act (20 U.S.C. 9622) (i.e., race, ethnicity, socioeconomic status, gender, disability, and limited English proficiency); and when addressing items in the criteria for student subgroups with respect to high school graduation rates, college enrollment and credit accumulation rates, and the assessments required under the ESEA, the State must provide data for the subgroups described in section 1111(b)(2)(C)(v)(II) of the ESEA (i.e., economically disadvantaged students, students from major racial and ethnic groups, students with disabilities; and students with limited English proficiency). We note that States are required under section 1111(b)(3)(C)(xiii) of the ESEA to also report achievement data disaggregated by gender and migrant status. Changes: As discussed earlier, we have added new paragraph (g) in the application requirements, which specifies that when addressing issues related to assessments required under the ESEA or subgroups in the selection criteria, the State must meet the following requirements: (1) For student subgroups with respect to the NAEP, the State must provide data for the NAEP subgroups described in section 303(b)(2)(G) of the National Assessment of Educational Progress Authorization Act (i.e., race, ethnicity, socioeconomic status, gender, disability, and limited English proficiency). The State must also include the NAEP exclusion rate for students with disabilities and the exclusion rate for English language learners, along with clear documentation of the State's policies and practices for determining whether a student with a disability or an English language learner should participate in the NAEP and whether the student needs accommodations; (2) For student subgroups with respect to graduation rates, college enrollment and credit accumulation rates, and the assessments required under the ESEA, the State must provide data for the subgroups described in section 1111(b)(2)(C)(v)(II) of the ESEA (i.e., economically disadvantaged students, students from major racial and ethnic groups, students with disabilities, and students with limited English proficiency); and (3) When asked to provide information regarding the assessments required under the ESEA, States should refer to section 1111(b)(3) of the ESEA; in addition, when describing this assessment data in the State's application, the State should note any factors (e.g., changes in cut scores) that would impact the comparability of data from one year to the next. Comment: One commenter recommended that the Department clarify that States must still meet AYP targets as required by the ESEA, even as they set new targets based on NAEP scores for Race to the Top accountability purposes. Another expressed concern that these criteria would tie State accountability goals and reporting to NAEP, which would conflict with ESEA requirements that link accountability to State-based standards and assessments. Discussion: The Department does not believe that additional language is required to clarify that States must still meet existing ESEA requirements. Neither the ARRA nor this final notice affects States' compliance with and obligations under the ESEA. Changes: None. B. Standards and Assessments Definitions: Common set of K-12 standards and high-quality assessment. Comments regarding the preceding definitions are addressed, as appropriate, below. Selection Criterion (B)(1): Developing and adopting common standards (Proposed Selection Criterion (A)(1)): Comment: Commenters were divided in their reactions to the criterion under which the Department would evaluate States' applications based on their commitment to adopt a common set of K-12 standards (as defined in this notice). Many commenters supported this criterion. Some suggested procedures that should be followed in the process of developing standards, including the need for broad participation from representatives of all student subgroups within a State prior to formal adoption of standards. A few commenters, however, were opposed to the adoption of common standards for various reasons, such as a lack of evidence that common standards will benefit students and the potential cost of adopting new content standards. One commenter urged removing participation in a consortium as a necessary condition of funding because of concerns that the size and the complexity of the relationships in a consortium may have the potential for conflicts of interest. Some commenters regarded the proposed criterion as punitive. A few commenters suggested making participation in common standards an invitational priority in the interest of making adoption truly voluntary. Another commenter expressed concern that a criterion under which States would be rewarded for their commitment for adopting a common set of K-12 standards will preempt what, up to now, has been a State-led process and would call into question the voluntary nature of State adoption of standards. Many commenters argued that States should be excused from the requirement to adopt common core standards if their current standards are as rigorous as common standards. One commenter suggested that the Department include in the final notice an additional criterion to provide recognition for those States with rigorous standards and improved student achievement. Another recommended an external review focused on rigor, college and career readiness and international benchmarking to determine whether adoption of a common set of K-12 standards is necessary. Discussion: We appreciate commenters' support for this criterion. The Department believes that States' adoption of common sets of K-12 standards will provide a foundation for more efficient and effective creation of the instructional and assessment resources needed to implement a coherent system of teaching and [[Page 59733]] learning. We do not agree that an external review is needed to determine whether States' adoption of common K-12 standards is necessary. Some readers appear to have been confused about the role of the criteria. One mistakenly believed that joining a consortium was a condition of funding under Race to the Top. This is not the case. Criteria are used to evaluate grant applications and applicants. States receive points for the strength and content of their responses to the criteria. In this program, we proposed that States' applications would be evaluated and receive points for demonstrating their commitment to improve standards by participating in a consortium of States working toward jointly adopting common K-12 standards. Thus, States with stronger proposals would receive more points; however, a State could receive a grant even without getting any points for this criterion. An individual State that chooses not to participate in a consortium for the development and adoption of common standards is eligible to apply for funds, but the application will not receive points under this criterion. A State that chooses not to join a consortium could describe its accomplishments in response to new criteria (F)(3) under which it could earn points for other significant reform conditions that have contributed to increased student achievement, narrowed achievement gaps, or other important outcomes. We decline to make participation in common standards an invitational priority for which a State would receive no points in the competition, rather than a selection criterion. We believe that common internationally benchmarked standards that prepare students for college and careers are a critical foundation for students' education and, therefore, are a component of a State's application deserving of evaluation and points in the competition. We agree that there is potential for conflicts of interest to arise within consortia, but believe there are ways for consortia to mitigate such conflicts and that removal of the criterion on these grounds is not warranted. Changes: None. Comment: Several commenters recommended that the Department clarify in the final notice whether the reference to common standards refers specifically to the common core standards currently being developed jointly by members of the National Governors Association and the Council of Chief State School Officers. Others recommended that the guidelines be modified to recognize other multi-State consortia that have defined or adopted common standards. One commenter requested recognition of the national collaborative of State leaders developing national standards and assessments in arts education. Discussion: In this program, the phrase ``common standards'' does not refer to any specific set of common standards, such as the common core standards currently under development by members of the National Governors Association and the Council of Chief State School Officers. The Department declines to make changes in order to endorse any particular standards-development consortium. Changes: None. Comment: Several commenters recommended that we clarify the meaning of ``a significant number of States'' within a consortium. One recommended that the number of States be set at a minimum of three if the quality of their common standards is comparable to the common standards developed by members of the National Governor's Association and the Council of Chief State School Officers. Others suggested that instead of a minimum number, the criterion should focus on the importance or potential impact of the proposed work. Discussion: The goal of common K-12 standards is to replace the existing patchwork of State standards that results in unequal expectations based on geography alone. Some of the major benefits of common standards will be the shared understanding of teaching and learning goals; consistency of data permitting research on effective practices in staffing and instruction; and the coordination of information that could inform the development and implementation of curriculum, instructional resources, and professional development. The Department believes that the cost savings and efficiency resulting from collaboration in a consortium should be rewarded through the Race to the Top program when the impact on educational practices is pronounced. And generally, we believe that the larger the number of States within a consortium, the greater the benefits and potential impact. We decline to define the term ``significant number of States'' by providing a particular number of States. We are providing additional information in Appendix B regarding how this selection criterion will be scored by reviewers and adding a cross reference to Appendix B in criterion (B)(1) to emphasize that States' evidence will be evaluated using Appendix B. Changes: The term ``significant number of States'' has been clarified in the Scoring Rubric (see Appendix B) so that, on this aspect of the criterion, a State will earn ``high'' points if its consortium includes a majority of the States in the country, and ``medium'' or ``low'' points if its consortium includes one-half of the States in the country or less. Additionally, we have added a reference to this in criterion (B)(1) by adding the parenthetical ``(as set forth in Appendix B)'' after ``evidenced by.'' Comment: Many commenters expressed concern regarding the proposed timeline for the adoption of common standards by June 2010. Commenters urged delay of the adoption target date in order to allow adequate time for activities such as local review and evaluation of the common standards, legislative or administrative action required for adoption, and broad stakeholder participation. Several pointed out that the proposed timeline for adoption of common standards by June 2010 conflicts with the timeline agreed to by governors and State chiefs currently participating in one consortium for the development of common standards. One commenter objected that the Race to the Top process does not allow States enough time to review the final standards from that consortium before submitting a grant application. Others questioned apparent differences for Phase 1 applicants and Phase 2 applicants regarding the actual adoption of common standards. Discussion: The Department recognizes that States need as much time as possible to review, evaluate, and adopt common K-12 standards. We are therefore extending the deadline for adopting standards as far as possible, while still allowing the Department to comply with the statutory requirement that the Department obligate all Race to the Top funds by September 30, 2010. The new deadline in this criterion for adopting common K-12 standards is August 2, 2010, or, at a minimum, by a later date in 2010 specified by the State. As described in the Scoring Rubric, States that meet the August 2, 2010 target date will earn more points for this criterion; a State that has a high-quality plan to adopt common standards by a later date in 2010 will earn some points for this criterion. In addition, we have clarified that Phase 1 applicants must demonstrate commitment to and progress toward adoption by August 2, 2010, or, at a minimum, by a later date in 2010 specified by the State, and that Phase 2 applicants must demonstrate adoption by that date in order to earn the most points for this criterion. We understand that adoption of standards is a legal process at the State level, and [[Page 59734]] fully expect that implementation of the standards will follow a thoughtful, deliberate course in subsequent year(s). For any State receiving funds, the Department will monitor the State's progress in meeting its goals and timelines as established in its plan, including States' progress towards adoption of common standards. Changes: We have revised the deadline in criterion (B)(1) regarding adoption of a common set of K-12 standards. Phase 1 applicants will be evaluated based on their high-quality plans demonstrating commitment to and progress toward adopting a common set of K-12 standards by August 2, 2010, or, at a minimum, by a later date in 2010 specified by the State. Phase 2 applicants will be evaluated based on whether they adopt such standards by August 2, 2010, or at a minimum, by a later date in 2010 specified by the State in a high-quality plan toward which the State has made significant progress. Both Phase 1 and Phase 2 applicants will also be evaluated on their commitment to implementing the standards after the deadline in a well-planned way. We also have revised and reorganized criterion (B)(1) non- substantively for purposes of clarity. When describing how a State can demonstrate its commitment to developing standards we have changed the phrase, ``improving the quality of its standards'' to ``adopting a common set of high-quality standards, as evidenced by * * *''. In criterion (B)(1)(ii)(a), we also have removed the qualifier to a common set of K-12 standards (``that are internationally benchmarked and that build toward college and career readiness by the time of high school graduation * * *'') because it is redundant with similar language in criterion (B)(1)(i)(a). Comment: Several commenters recommended that the Department clarify in the final notice the evidence necessary for criterion (B)(1), asking whether participation in a standards development consortium or an expression of intent to participate in such a consortium, such as a Memorandum of Agreement, is sufficient. One commenter suggested that a State should be allowed to provide whatever evidence it believes is appropriate to demonstrate its efforts to address this criterion. Discussion: We agree that the evidence for this criterion should be clearer, and have made some revisions to the evidence requested for that purpose. The evidence requested is shown in Appendix A of this notice. We do not agree with the commenter that a State should provide whatever evidence it believes is appropriate to demonstrate its efforts to address this criterion. Changes: We have clarified some of the requested evidence for criterion (B)(1). We request that a State supply a copy of the Memorandum of Agreement, executed by the State, showing that it is part of a standards consortium, and provide the number and names of States participating in the consortium. A State should provide a copy of the final standards, or if the standards are not yet final, a copy of the draft standards and anticipated date for completing the standards. A State should also provide documentation that the standards are or will be internationally benchmarked. For Phase 1, States must provide a description of the legal process in the State for adopting standards, and the State's plan, current progress, and timeframe for adoption. For Phase 2, States must show evidence that they have adopted the standards; or, if the State has not yet adopted the standards, provide a description of the legal process in the State for adopting standards, and the State's plan, current progress, and timeframe for adoption. States may provide additional evidence beyond that requested. Comment: One commenter asked what national and international benchmarks are required under criterion (B)(1). Discussion: The Department is not requiring that common standards adopted by State applicants be benchmarked to particular international standards, but the standards should be supported by evidence that they are internationally benchmarked. Changes: We have revised criterion (B)(1)(i) to clarify that the K- 12 standards adopted by the State should be ``supported by evidence that they are'' internationally benchmarked. Comment: A few commenters requested more detail regarding the desired characteristics of college and career ready standards. Some suggested that the Department require specific types of evidence to meet this criterion, such as measurement of the skills needed to succeed in non-remedial college courses, validation by the postsecondary system or involvement of postsecondary faculty in development of the standards and assessments. Discussion: Criterion (B)(1) focuses on States' development and adoption of common K-12 standards that build toward college and career readiness. By using these terms, we mean that the standards should build on content knowledge and skills regarded as essential for success in college and the workforce. The Department recognizes that many kinds of documentation could reasonably support the claim that common standards build toward college and career readiness and prefers to leave the selection of appropriate documentation to the States. Changes: None. Definition of Common Set of K-12 Standards Comment: We received several recommendations to modify the definition of common set of K-12 standards. Some commenters suggested that the definition of common set of K-12 standards should refer to 21st century skills; English language proficiency standards aligned to the language arts standards; and standards for science, technology, and engineering. Another commenter recommended expanding the definition to include standards currently shared across States, such as the American Diploma Project standards or ACT College Benchmarks. Other commenters recommended that the definition clearly specify whether the common standards should include standards for each high school grade or for each high school course. One commenter asked if the term ``standard'' refers to a broad statement about content or to a discrete concept or skill. Discussion: It is up to States participating in the development of common standards to determine the content and scope of the standards, whether to organize the standards for high school by grade or by course, and whether the statement of each standard is focused broadly on general concepts or narrowly on particular skills. Therefore, we decline to make the changes recommended by the commenters. Changes: None. Comment: Commenters requested clarification of what it means for the common standards to be ``identical'' across all States in a consortium given that a State may supplement the common standards with additional standards. Some commenters suggested changing the definition to refer to standards that are ``aligned,'' across States, rather than ``identical.'' Other commenters suggested that the additional standards adopted by a State should be more stringent than the common standards, foster innovation, or focus on particular skills of local relevance. Discussion: Some commenters appeared to be confused by the term ``identical'' when it was qualified by the possible addition of a supplementary group of standards that could vary across States in a consortium. The term [[Page 59735]] ``identical'' refers to the common standards and not the supplementary standards and would not permit the standards to be ``aligned'' across States in a consortium, as recommended by one commenter. Upon further reflection, we believe that there may be reasons for the common standards to be slightly different across States (e.g., States may use slightly different terms to refer to the same concepts or may have a particular format which would require slight changes in language) and therefore, are changing ``identical'' to ``substantially identical.'' The Department believes that it is unnecessary to include in the definition additional requirements for the supplementary standards, such as being more rigorous or fostering innovation, and therefore, declines to change the definition as requested by commenters. Changes: We have changed ``identical'' to ``substantially identical'' to clarify that a common set of K-12 standards are ``substantially identical'' across all States in a consortium. Selection Criterion (B)(2): Developing and Implementing Common, High-quality Assessments (Proposed Selection Criterion (A)(2)): Comment: A number of commenters supported the Department's proposal to evaluate a State's commitment to improving the quality of its assessments by participating in a consortium of States developing common high-quality assessments (as defined in this notice) aligned with the consortium's common set of K-12 standards. However, other commenters requested that the Department remove this criterion, stressing that the Department has overemphasized standardized testing and that the ESEA has stressed reading and math to the detriment of other subjects. One commenter asserted that a State should not have to join a consortium if its own assessment is of high quality. Another commenter questioned why we would encourage States to change current assessment programs; this commenter suggested that we not replace current assessments until there is certainty about which aspects of current testing need change so as to not waste resources and risk development of low-quality assessments. Another commenter suggested the Department support the improvement of State and local assessment systems rather than pressuring States to ``swap one standardized test for another.'' Discussion: The Department believes that consortia of States, by pooling resources, will be able to produce significantly higher-quality assessments more cost-effectively than any one State could produce alone. Significant improvement of student outcomes can be realized when high-quality assessments aligned to common standards inform and support teacher instruction and, thus, student learning. An individual State that chooses not to participate in a consortium for the development and adoption of assessments aligned to common standards is eligible to apply for funds, but the application will not receive points for this criterion. We understand commenters' concerns about the overemphasis of standardized testing, but believe that educators need good information about what students know and can do so that they can guide their students' learning, and adjust and differentiate their instruction appropriately. This information needs to come, in part, from academic assessments. With respect to support for local assessments, criteria (B)(3) and (C)(3) provide opportunities for focus on local assessments and instructional improvement systems. Criterion (B)(3) evaluates a State on the extent to which it has a high-quality plan for supporting statewide transition to and implementation of enhanced standards and high-quality assessments and provides examples of State or LEA support activities, including implementation of high-quality instructional materials and assessments. In responding to this criterion, States could propose to support development of local assessments, including formative and interim assessments, that would assist in the transition to new statewide standards and assessments. Criterion (C)(3) evaluates a State on the extent to which it has a high-quality plan to increase the acquisition, adoption, and use of local instructional improvement systems (as defined in this notice); supports LEAs and schools that are using instructional improvement systems; and makes data from these systems available and accessible to researchers. Instructional improvement systems may include local assessment data. Changes: None. Comment: Several commenters opposed the provision in criterion (B)(2) that asked a State to describe the extent to which its consortium working on developing common high-quality assessments includes a ``significant number of States,'' recommending instead that the criterion focus only on the quality of the assessments. One commenter recommended that the criterion evaluate the extent to which the consortium has the potential to have a significant national impact, including consideration of the number and diversity of students in participating States, or the ability of participating States to serve as exemplars for statewide reform, rather than focus on the number of participating States. Discussion: The Department believes that the cost savings and efficiency resulting from collaboration in a consortium should be rewarded through Race to the Top when the impact on educational practices is pronounced. Generally, we believe that the larger the number of States within a consortium, the greater the benefits and potential impact. While the other measures suggested by the commenters could be valuable, they would not be as objective a measure for the reviewers to consider when evaluating a State's plan. We are providing information about the scoring of this criterion in the Scoring Rubric set forth in Appendix B. Additionally, we are adding a cross reference to Appendix B in criterion (B)(2) to emphasize that States' evidence will be evaluated using Appendix B. Changes: The term ``significant number of States'' has been clarified in the Scoring Rubric (see Appendix B) so that, on this aspect of the criterion, a State will earn ``high'' points if its consortium includes a majority of the States in the country, and ``medium'' or ``low'' points if its consortium includes one-half of the States in the country or less. Additionally, we added the parenthetical ``(as set forth in Appendix B)'' after ``evidenced by'' in criterion (B)(2). In addition, we have made some non-substantive changes to this section for clarity. We have replaced ``whether'' with ``to the extent to which'' in criterion (B)(2); we have added ``as evidenced by (i) the State's participation * * *''; and we have removed the phrase ``that are internationally benchmarked'' when we refer to a common set of K-12 standards because the phrase is unnecessary and redundant with language in criterion (B)(1)(i)(a). Comment: Many commenters suggested that the Department consider additional factors in examining a State's commitment to developing common assessments. One commenter recommended that States submit evidence from assessment developers demonstrating that the assessments are valid and reliable for English language learners, as well as showing the research base for use of accommodations. Another commenter suggested that the criterion explicitly encourage States to develop a more comprehensive local assessment system. [[Page 59736]] Discussion: Members of an assessment consortium are responsible for ensuring that assessments are developed to meet the definition of high- quality assessments (as defined in this notice), including the requirement that assessments are of high technical quality and include students with disabilities and English language learners. Local assessments can be addressed in response to other criteria, such as criterion (B)(3) and (C)(3) as previously discussed. Changes: None. Comment: A number of commenters requested that the Department clarify in the final notice how an applicant should describe its strategy for and commitment to joining a common assessments consortium and implementing common assessments. One commenter suggested that States demonstrate compliance with this criterion by developing a timeline for when assessments would be aligned to the common standards. Two commenters asked if States can include the cost of additional assessments, such as formative and benchmark assessments, in addition to summative tests in its application. Another commenter suggested that we evaluate States' progress in relation to developing common assessments on a regular basis and that reports should be provided on these evaluations. Discussion: It is not necessary for a State to describe its strategy for joining a common assessments consortium; the evidence for this criterion focuses on a State's participation in a consortium that intends to develop high-quality assessments. The minimum evidence for which a State will receive points for this criterion is described in detail in Appendix A of this notice (Evidence and Performance Measures). The Department intends to hold a separate Race to the Top Assessment competition that will fund the development of common, summative assessments tied to common K-12 standards. We therefore believe that funds within this Race to the Top competition would be better spent on other activities. Accordingly, we have added a requirement specifying that no funds awarded under this competition may be used to pay for costs related to statewide summative assessments. Formative and interim assessments (as defined in this notice) may be funded within this competition, and would be funded as part of a State's plan for criterion (B)(3). In addition, for any State receiving funds, the Department will monitor the State's progress in meeting its goals and timelines. Changes: We have added a program requirement that no funds awarded under this competition may be used to pay for costs related to statewide summative assessments. Comment: A few commenters suggested that high-quality assessments include grade-by-grade specificity of core subject matter. Others suggested this notice explicitly include the assessment of broad-based humanities centered curricula, including art, science, and social studies. Discussion: This notice does not limit or require certain grade or content coverage for high-quality assessments. Changes: None. Comment: Another commenter suggested that we award additional points to States that commit to developing a common STEM assessment. Discussion: A State may choose to address competitive preference priority 2, which addresses STEM issues, and, if peer reviewers determine the State has met the priority, would receive extra points in the Race to the Top competition. The third element of this priority (a plan to address the need to prepare more students for advanced study and careers in the sciences, technology, engineering, and mathematics) could be addressed, in part, by a commitment to develop a common STEM assessment. Note, however, that a statewide summative STEM assessment would have to be developed using funds other than those awarded under this competition because, as noted in the previous comment, Race to the Top funds cannot be used to pay for costs related to statewide summative assessments. Changes: None. Comments: Some commenters asked that the Department provide incentives for States to develop and implement high-quality assessments beginning at pre-kindergarten. Discussion: As previously stated, this notice does not limit or require certain grade or content coverage for high-quality assessments. We note, however, that invitational priority 3 invites States to include in their applications practices, strategies, or programs to improve educational outcomes for high-need young children by enhancing the quality of preschool programs. Of particular interest are proposals that support practices that (i) improve school readiness (including social, emotional, and cognitive); and (ii) improve the transition between preschool and kindergarten. Changes: None. Comment: One commenter requested that the Department state in the final notice that new assessment systems should be aligned with content standards, and be vertically integrated. Another commenter suggested that the entire K-12 assessment system should be vertically moderated to the anchor assessments so ``proficient'' means ``prepared'' and that students are on-track to meet college and career ready standards by graduation. Discussion: Under criterion (B)(2) States will be rewarded for the development of assessments aligned with common standards that build toward college and career readiness. The technical aspects of how the assessment system is organized to reflect increasing student competence from grade to grade will be determined by the consortia developing the assessments. Changes: None. Comment: One commenter stated that a plan for implementing high- quality assessments must include high-quality alternate assessments. Discussion: We agree with the commenter; however, we do not believe it is necessary to include additional language to that effect in this notice because section 1111(b)(3)(C)(ix)(II) of the ESEA requires that States include students with disabilities in their assessments. In addition, section 612(a)(16)(C) of the IDEA requires States to provide an alternate assessment to a student with a disability who needs it for any statewide assessment. Changes: None. Comment: Several commenters supported the statement in the NPP that, at a later date, we may announce a separate Race to the Top Assessment Competition, for approximately $350 million, to support the development of assessments by consortia of States. Several commenters asked for more explicit guidelines on standards and assessment work for Phases 1 and 2 as described in this notice, as opposed to the work for the separate $350 million fund for the development of assessments. Discussion: As previously indicated, the Department intends to hold a separate Race to the Top Assessment competition that will fund consortia in developing common, summative assessments tied to common K- 12 standards. The Department may provide additional information about this competition in the future, and as noted previously, more requirements may be articulated in that competition's notice. Changes: None. Definition of High-Quality Assessment Comment: Many commenters supported the proposed definition of [[Continued on page 59737]] From the Federal Register Online via GPO Access [wais.access.gpo.gov] ] [[pp. 59737-59786]] Race to the Top Fund [[Continued from page 59736]] [[Page 59737]] high-quality assessment. Several commenters recommended that the definition refer to the use of universal design principles in test development and administration. A few commenters suggested revising the definition to clarify that the use of open-ended items, performance- based tasks, and technology are desirable and necessary only insofar as they are grade-appropriate for the subject matter and consistent with the skills to be measured. Many other commenters recommended revising the definition to include assessments and assessment systems that measure higher order and critical thinking, problem-solving, reasoning, research, writing, scientific investigation, communication, and teamwork skills. Discussion: We agree with the commenters that the definition of high-quality assessment should refer to the use of universal design principles in test design and administration and are making that change. However, we are not revising the definition to include specific skills, such as critical thinking, problem solving, research, or writing skills, mentioned by the commenters because the skills and content included in an assessment will be determined by the content standards on which the assessment is based. Instead, we are revising the definition to state that a high-quality assessment is an assessment that is designed to measure a student's ``knowledge, understanding of, and ability to apply, critical concepts,'' rather than an assessment that is designed to measure ``understanding of, and ability to apply, critical concepts.'' We do not believe it is necessary to clarify that open-ended items, performance-based tasks, and technology should be appropriate for the grade and subject to be assessed and consistent with the skills to be measured, as recommended by commenters. We believe this is implicit in the design of any assessment and have included open-ended responses, performance-based tasks, and technology as examples, not as requirements of a high-quality assessment. Finally, based on the Department's internal review, we are making several changes to the definition. First, in the NPP, we stated that a high-quality assessment uses a ``variety of item types, formats, and administration conditions (e.g., open-ended responses, performance- based tasks, technology).'' We believe that a variety of administration conditions is not necessarily a requirement for an assessment to be of high quality. Therefore, we are revising the definition to clarify that a high-quality assessment uses a variety of item types and formats (e.g., open-ended responses, performance-based tasks) and incorporates technology, where appropriate. Second, for consistency with the rest of the notice, we are changing the reference to ``limited English proficient students'' to ``English language learners.'' Next, the proposed definition stated that a high-quality assessment be ``of high technical quality (e.g., valid, reliable, and aligned to standards).'' For completeness, we are adding ``fair'' to the examples in the parenthetical. Finally, for clarity, we are changing ``Such assessments are structured to enable measurement of student achievement * * *'' to ``Such assessments should enable measurement of student achievement.'' Changes: With the aforementioned changes, the definition of high- quality assessment is as follows: ``High-quality assessment means an assessment designed to measure a student's knowledge, understanding of, and ability to apply, critical concepts through the use of a variety of item types and formats (e.g., open-ended responses, performance-based tasks). Such assessments should enable measurement of student achievement (as defined in this notice) and student growth (as defined in this notice); be of high technical quality (e.g., be valid, reliable, fair, and aligned to standards); incorporate technology where appropriate; include the assessment of students with disabilities and English language learners; and to the extent feasible, use universal design principles (as defined in section 3 of the Assistive Technology Act of 1998, as amended, 29 U.S.C. 3002) in development and administration.'' Comment: Several commenters recommended that the Department require that high-quality assessments address the needs of English language learners, students with disabilities, and other learners who need targeted services. Discussion: As defined in this notice, a high-quality assessment includes assessment of students with disabilities and English language learners. Changes: None. Selection Criterion (B)(3): Supporting the Transition to Enhanced Standards and High-Quality Assessments (Proposed Selection Criterion (A)(3)): Comment: Many commenters approved of criterion (B)(3) regarding a State's high-quality plan for supporting a statewide transition to and implementation of enhanced standards and high-quality assessments, but stated that the Department should expand the activities that a transition plan might include. For instance, several commenters suggested that States show that they plan to increase student participation in Advanced Placement and International Baccalaureate courses, as well as dual enrollment in postsecondary credit-bearing courses, while transitioning to common standards and assessments. A few commenters suggested States commit to increasing student participation in pre-Advanced Placement courses for middle school students, and in after-school programs to accelerate achievement for students having difficulty meeting academic targets. One commenter recommended that States provide a roll-out plan for adoption of the common standards and all of their supporting components. Some commenters suggested that adoption of common standards be accompanied by the necessary supporting components, such as curricular frameworks, unit plans, lesson plans, curriculum-embedded formative assessments, anchor assignments, and rubrics. One commenter noted that States should amend course requirements for graduation to ensure that students are guaranteed to receive the content. However, not all commenters supported additional supports and resources during a State's transition to enhanced standards and high- quality assessments. One commenter questioned whether limited Race to the Top funds should be used by States and LEAs to develop instructional materials. Another commenter was critical of requiring a plan for transition; instead this commenter suggested that a State should be judged on its transition after implementation of common standards and assessments, not before the State has developed best practices. Discussion: We agree with many of the commenters' suggestions regarding which supporting components should be considered when transitioning to new standards and assessments. We encourage States to create plans that increase student participation in advanced coursework in order to provide for a smooth transition to internationally benchmarked standards aligned with college and career ready expectations. We also agree that a rollout plan and additional supports would aid in the transition to enhanced standards and high-quality standards, and have therefore incorporated these suggestions. We understand the commenter's concern that States may need to amend course requirements for graduation to ensure that students are guaranteed to receive the content. We [[Page 59738]] believe a statement in criterion (B)(3) addresses this comment--that State or LEA activities might include, ``in cooperation with the State's institutions of higher education, aligning high school exit criteria and college entrance requirements with new standards and assessments.'' We disagree with commenters who questioned whether limited Race to the Top funds should be used by States and LEAs to develop instructional materials. We believe that the transition to enhanced assessments and a common set of K-12 standards will not be successful without support from the States doing this work in collaboration with their participating LEAs. We have made several edits for clarity in the illustrative list of State and LEA support activities for transition to enhanced standards and high-quality assessments. We deleted the reference to developing curricular frameworks, for example, but added a reference to ``high- quality instructional materials and assessments (including, for example, formative and interim assessments).'' Additionally, we accepted commenters' suggestion to add ``development of a rollout plan for the standards with all supporting components,'' which could include, among other things, development of curricular frameworks and materials. Changes: We have revised the language in criterion (B)(3) to include many of the commenters' suggestions. The language now reads that State or LEA activities might, for example, include, ``developing a rollout plan for the standards together with all of their supporting components; in cooperation with the State's institutions of higher education, aligning high school exit criteria and college entrance requirements with the new standards and assessments; developing or acquiring, disseminating, and implementing high-quality instructional materials and assessments (including, for example, formative and interim assessments (both as defined in this notice)); developing or acquiring and delivering high-quality professional development to support the transition to new standards and assessments; and engaging in other strategies that translate the standards and information from assessments into classroom practice for all students, including high- need students (as defined in this notice).'' Comment: One commenter suggested including, as an additional activity to support statewide transition to and implementation of enhanced standards and high-quality assessments, building improvements for science labs and technology in the classrooms. Discussion: Consistent with the Department's May 11, 2009, State Fiscal Stabilization Fund guidance ,\3\ the Department also discourages States and LEAs from using Race to the Top funds for new construction because this use may limit the ability of the State and its LEAs to implement the State's core Race to the Top plans. States may propose that certain participating LEAs may use Race to the Top funds for modernization, renovation, or repair projects to the extent that these projects are consistent with the State's Race to the Top plan. --------------------------------------------------------------------------- \3\ Available at: http://www.ed.gov/programs/statestabilization/guidance-mod-05112009.pdf. --------------------------------------------------------------------------- Changes: None. Comment: Several commenters observed that teachers will be primarily responsible for ensuring successful implementation of new standards and, accordingly, recommended that teachers be involved in a State's transition plan. Commenters stated that a transition plan should include model lesson plans, pre-service teacher education, and in-service professional development to familiarize and train teachers on the content standards and how to use assessment results. One commenter suggested that professional development be focused on middle school and high school teachers. Discussion: We agree with commenters that a successful transition plan should include high-quality professional development to support the transition to new standards and assessments. The NPP included developing, disseminating and implementing professional development materials as a suggested State or local activity in this criterion. We are strengthening the language about this activity to suggest development or acquisition and delivery of high-quality professional development to support the transition to new standards and assessments. We also agree with the commenter that teachers should be involved in a State's transition plan. Under criterion (B)(3) the Department will evaluate a State application on the extent to which it has a high- quality plan for supporting the transition to and implementation of enhanced standards and high-quality assessments, in collaboration with its participating LEAs. We expect that LEAs will collaborate with teachers on this criterion. In addition, in criterion (A)(2)(ii)(a), a State is judged on the extent to which it has a high-quality overall plan to (among other things) utilize the support it has from a broad group of stakeholders to better implement its plans, as evidenced by the strength of the statements or actions of support from the State's teachers and principals, which include the State's teachers' unions or statewide teacher associations. We decline to take the commenter's suggestion that a State focus its professional development on middle and high school teachers because we believe all teachers implementing enhanced standards and high- quality assessments would benefit from high-quality professional development. Changes: We have included language in criterion (B)(3) to clarify that a State or LEA activity might, for example, include ``developing or acquiring and delivering high-quality professional development to support the transition to new standards and assessments.'' Comment: Numerous commenters articulated a need for collaboration, stakeholder engagement, financial support, autonomy, and flexibility during the transition to enhanced standards and assessments. One commenter stated that unless States are committed to the adoption and implementation of the standards, and support LEAs and schools in implementing them, the new standards and assessments will not positively affect teaching or learning. One commenter suggested that the State plans require local school boards to ensure collaboration between school administrators and union leaders to ensure that all educators are part of the alignment of assessments. A few commenters urged the Department to encourage continuity between pre-kindergarten and elementary school as part of the transition process. One commenter supported efforts to promote a seamless articulation of standards and assessments between pre-kindergarten, K-12, and post-secondary education, since any gap leads to critical loss of learning for students. Discussion: The Department agrees with commenters that collaboration, support, and engagement are critical factors for a successful transition to enhanced standards and high-quality assessments. The criteria in (A) establish State Success Factors, which ask States to articulate their education reform agendas and LEAs' participation in it, and explain their strategies for building strong statewide capacity to implement, scale and sustain proposed plans. Specifically, criterion (A)(2)(ii) provides for evaluation of a State's plan to utilize the support it has from a broad group of stakeholders to better implement its plans, as evidenced by [[Page 59739]] the strength of statements or actions of support from critical stakeholders. Changes: None. Comment: One commenter requested clarification about whether all LEAs or only participating LEAs must transition to the enhanced standards and high-quality assessments. Many commenters noted that the adoption of common standards will affect all LEAs, not only those participating in a State's Race to the Top application. Accordingly, commenters suggested that a State include in its plan how it will provide direct financial support for the operational costs incurred by LEAs as they transition to common standards and assessments. Discussion: The NPP was clear that a State will be judged on the extent to which it has a high-quality plan for supporting a statewide transition to a common set of K-12 standards and high-quality assessments aligned to those standards. We recognize that a statewide system of standards and assessments eventually would be implemented in all LEAs, some of which are not participating in the Race to the Top grant. To address this situation, we are adding a new definition of involved LEAs. An involved LEA is an LEA that chooses to work with the State to implement those specific portions of the State's plan that necessitate full or nearly-full statewide implementation, such as transitioning to a common set of K-12 standards. Involved LEAs do not receive a share of the 50 percent of a State's grant award that it must subgrant to LEAs in accordance with section 14006(c) of the ARRA, but States may provide other funding to involved LEAs under the State's Race to the Top grant in a manner that is consistent with the State's application. We expect that participating LEAs will have a greater role than involved LEAs in collaborating with States as States develop their plans, but believe that the specifics of such decisions are best left to local decision makers. Changes: We have added a new definition of involved LEAs, which reads as follows: ``Involved LEAs mean LEAs that choose to work with the State to implement those specific portions of the State's plan that necessitate full or nearly-full statewide implementation, such as transitioning to a common set of K-12 standards (as defined in this notice). Involved LEAs do not receive a share of the 50 percent of a State's grant award that it must subgrant to LEAs in accordance with section 14006(c) of the ARRA, but States may provide other funding to involved LEAs under the State's Race to the Top grant in a manner that is consistent with the State's application.'' Comment: One commenter recommended that States should provide minimum protections for their students during the transition to new standards and assessments, including a period of time to orient students and teachers to new standards and assessments, to ensure instruction time, and to eliminate disparate impact on minority students. One commenter requested that the Department address equity in the adequacy of instructional materials, suggesting that States ensure that every student has access to print or digital instructional materials that are current and aligned to the enhanced standards. Discussion: We agree with commenters that a State should address supports for high-need students in its plan to transition to enhanced standards and high-quality assessments. We are adding a reference to high-need students in criterion (B)(3) and including a definition of high-need students in the Definitions section of this notice. States should have the flexibility to decide on the appropriate supports for their high-need students; therefore, we decline to specify the supports States must provide to students. Changes: We have added language to criterion (B)(3) indicating that State or LEA activities might include ``engaging in other strategies that translate the standards and information from assessments into classroom practice for all students, including high-need students (as defined in this notice).'' We also have added a definition of high-need students, which reads as follows: ``High-need students means students at risk of educational failure or otherwise in need of special assistance and support, such as students who are living in poverty, who attend high-minority schools (as defined in this notice), who are far below grade level, who have left school before receiving a regular high school diploma, who are at risk of not graduating with a diploma on time, who are homeless, who are in foster care, who have been incarcerated, who have disabilities, or who are English language learners.'' Comment: One commenter suggested that a State demonstrate that its public higher education institutions will certify readiness for entry into credit-bearing coursework if students meet the high school common standards through completing a course of study aligned with those standards and score at the defined college-ready level on high school assessments. Discussion: We do not believe that we should prescribe the exact policy mentioned by the commenter; we believe a State should have the flexibility to determine, in cooperation with its institutions of higher education, the best way to align high school exit criteria and college entrance requirements with the new standards and assessments. However, we believe that some clarification of the language in criterion (B)(3) is necessary and have revised accordingly. Changes: Criterion (B)(3) has been revised to provide that State or LEA activities might, for example, include, ``in cooperation with the State's institutions of higher education, aligning high school exit criteria and college entrance requirements with the new standards and assessments.'' Comment: A few commenters requested that States provide minimum evidence as to how they are ensuring proper implementation of their current standards, including evidence of actual implementation in classrooms, such as survey results from a representative sample of teachers demonstrating how standards are being disseminated and utilized. Discussion: For any State receiving funds, the Department will monitor the State's progress in meeting its goals and timelines as established in its plan. Rather than requiring a State to use survey results as minimum evidence for this criterion, as some commenters suggested, we will be gathering this kind of information through evaluations. As stated elsewhere in this notice, IES will be conducting a series of national evaluations of Race to the Top State grantees as part of its evaluation of programs funded under the ARRA. Race to the Top grantee States are not required to conduct independent evaluations, but may propose, within their applications, to use funds from Race to the Top to support independent evaluations. Grantees must make available, through formal or informal mechanisms, the results of any evaluations they conduct of their funded activities. In addition, as described elsewhere in this notice and regardless of the final components of the national evaluation, Race to the Top States, LEAs, and schools are required to make work developed under this grant freely available to others, and should identify and share promising practices and make data available to stakeholders and researchers (in appropriate ways that must comply with FERPA, including 34 CFR Part 99, as well as State and local requirements regarding privacy). Changes: None. [[Page 59740]] C. Data Systems To Support Instruction Definitions: Instructional Improvement System Comments regarding the preceding definition are addressed, as appropriate, below. Selection Criterion (C)(1): Fully Implementing a Statewide Longitudinal Data System (Proposed Selection Criterion (B)(1)): Comment: Many commenters supported criterion (C)(1) that provides for a State to be evaluated based on the extent to which it has a statewide longitudinal data system that includes all of the America COMPETES Act elements. Other commenters suggested that the Department consider using Race to the Top funds for purposes other than data systems, such as providing direct services in schools with demonstrated needs or improving the infrastructure for the delivery of instruction. One commenter suggested using the funds to develop new standards and assessments first, rather than building a longitudinal data system based on current standards and assessment systems. One commenter suggested that rather than having a major focus on State collection and sharing of data, the Department should require States to help schools and LEAs develop longitudinal data collection systems. Discussion: The Department appreciates the support for the development and implementation of statewide longitudinal data systems. We disagree with commenters who recommend that funds not be used for this purpose. Data is an important tool to identify needs and improve instruction. In addition, section 14006(a)(2) of the ARRA directs the Secretary to make grants to States that have made significant progress in meeting the objectives of paragraphs (2), (3), (4), and (5) of section 14005(d), including the development of statewide longitudinal data systems that include the elements described in section 6401(e)(2)(D) of the America COMPETES Act. While criterion (C)(1) is a measure of the current status of States' implementation of their statewide longitudinal data systems under the America COMPETES Act (as defined in this notice), both criteria (C)(2) and (C)(3) provide for the evaluation of States' plans to enhance their statewide longitudinal data systems and local instructional improvement systems. Funds awarded under the Department's statewide longitudinal data systems grants program may also be used, in coordination with Race to the Top funds, to build out a State's data infrastructure. Changes: None. Comment: One commenter indicated that a State should plan for the operational costs of implementing data systems that a Race to the Top grant does not cover. This commenter recommended that the Department require each State to specifically indicate in its application how it plans to technically and financially support LEAs across the State, including developing contracts and systems that can reduce costs by involving multiple LEAs. Discussion: We agree with commenters that it is important for a State to consider funding issues in its data system implementation plans, as well as its overall plans. Under criterion (A)(2)(i)(e), a State will be evaluated on the extent to which it has a high-quality overall plan to ensure that it has the capacity required to implement its proposed plans by using the fiscal, political, and human capital resources of the State to continue, after the period of funding has ended, those reforms funded under the grant for which there is evidence of success. Changes: None. Comment: Many commenters applauded criterion (C)(1), which evaluates the extent to which a State has a statewide longitudinal data system that includes all of the elements specified in section 6401(e)(2)(D) of the America COMPETES Act. Several commenters specifically highlighted the importance of including unique identifiers for students, teachers, and administrators in the list of America COMPETES Act data elements. However, many commenters suggested additional data elements that should be collected and reported through these systems. Commenters indicated that these data systems should include multiple achievement measures and multiple data sources, such as annual achievement data for all core academic subjects, as defined in the ESEA, valid and reliable local and State assessment results, formative assessment results, performance assessment results, and English language proficiency results. One commenter recommended that the data systems include data that demonstrate a student's ability to apply, analyze, synthesize and evaluate content knowledge. A few commenters recommended collecting data on the rates of students reading at grade- level by grade 3. Some commenters recommended various ways data should be disaggregated. They suggested that statewide longitudinal data systems be designed to allow for analysis of student achievement by race, ethnicity, socioeconomic status, gender, disability, and English language learner status. One commenter recommended that the Department encourage States to disaggregate data of vulnerable populations such as pregnant and parenting students. One commenter noted that it is critical that the statewide longitudinal data system measure all proficiency levels (i.e., below proficiency, at proficiency, above proficiency, and advanced) instead of just measuring below or above proficiency. Other commenters recommended non-assessment related data elements to be included in statewide longitudinal data systems, such as college readiness indicators, graduation rates, attendance rates, student enrollment data, course enrollment, student mobility rates, budget information, completion rates, curriculum changes, and instructional time. A few commenters suggested that in order to evaluate the progress of individual students through the K-12 system and into postsecondary education, systems should include information such as the percentage of students from each high school enrolling in institutions of higher education, students taking remedial or developmental coursework in college, or the points at which students exit, transfer in, transfer out, drop out, or complete P-16 education programs. One commenter suggested that the data systems include model lesson plans for teachers. Some commenters recommended including data related to indicators of school safety, culture and climate. Others suggested including information about student, family and community engagement. A few commenters requested that the data systems include student social service-related data elements and health indicators, such as immunization rates, asthma rates, vision and hearing screening, and obesity rates. Several commenters recommended including measures of students' social and emotional health and character development. Others believed that data systems should provide data regarding the numbers of transfers, dropout rates, chronic absenteeism, suspension rates, truancy, and dropout re-enrollment in order to trigger supports and interventions for students and families. Commenters also suggested that statewide longitudinal data systems should include data about teaching and learning conditions, such as teacher recruitment and retention, educator turnover, pupil and teacher ratios, [[Page 59741]] subject area teacher certification, full-time equivalent teacher employment, and the commitment to current educational programs (i.e., whether the curriculum has changed) in order to help schools, districts and States better understand supports and barriers to teacher effectiveness. One commenter recommended that statewide longitudinal data systems include information about English language learners, such as the type of English language learner instructional program in which a student participates, time in that program, level of English proficiency, and date of reclassification. Some commenters suggested requiring data about student participation in other programs, such as data on students served in gifted and talented education programs, innovative programs, expanded learning programs, or students receiving advanced coursework. One commenter recommended that data on technology use be explicitly included in statewide longitudinal data systems. Some commenters recommended that statewide longitudinal data systems include linkages with students in adult basic education, workforce and skills training programs and corrections systems, and student information from State employment wage records. One commenter stated that we did not provide sufficient justification for why all these data elements are essential. Another commenter suggested that the Department give States latitude to define the elements included in their data systems. Discussion: Some of the data elements suggested by commenters mirror the data elements listed in the America COMPETES Act. Although the Department will not be evaluating whether a State's system has information beyond the 12 elements of the America COMPETES Act, we recognize the varying needs and capabilities of States, and we encourage States to track additional information through their longitudinal data systems or to add additional components to their State plans to the extent the State deems appropriate. However, the Department recognizes the financial burden of collecting data, and we believe that it is sufficient to specifically evaluate States only on the extent to which their statewide longitudinal data systems include the elements in the America COMPETES Act. As stated in invitational priority 4, the Secretary is particularly interested in applications in which the State plans to expand statewide longitudinal data systems to include or integrate data from special education programs, English language learner programs, early childhood programs, at-risk and drop-out prevention programs, and school climate and culture programs, as well as information on student mobility, human resources (i.e. information on teachers, principals, and other staff), finance, student health, postsecondary, and other relevant areas, with the purpose of connecting and coordinating all parts of the system to allow important questions related to policy, practice, or overall effectiveness to be asked and answered, and incorporated into effective continuous improvement practices. While the Secretary is interested in applications that meet this invitational priority, a State meeting the priority would not receive additional points or preference over other applications. A State will be evaluated based on the extent to which it has a statewide longitudinal data system that includes all of the elements specified in section 6401(e)(2)(D) of the America COMPETES Act. Changes: None. Early Childhood Comment: Several commenters supported the fact that a statewide longitudinal data system, as specified by the America COMPETES Act, would include student information beginning at the pre-kindergarten level. Some commenters recommended that the Department require a State to expand its longitudinal data system by linking with available data on young children; their participation in early childhood education programs; and the characteristics, quality, staffing, and funding of those programs in order to increase access, improve quality, identify critical social services and interventions, and align standards, curricula and assessments from pre-kindergarten through grade 3. A few commenters recommended that a data system be designed so that data eventually can be captured at birth and fed into a Quality Rating Improvement System, if a State has such a system. Discussion: We agree with commenters that data about early childhood education programs are important to help ensure that young children begin school ready to learn. The America COMPETES Act elements specify a pre-kindergarten-16 data system. If it chooses, a State may link its longitudinal data system to available data on young children and their participation in early childhood programs, consistent with FERPA, including 34 CFR Part 99. This notice has several invitational priorities regarding early childhood programs: (a) Invitational priority 3, inviting applications in which the State plans to create practices, strategies, or programs to improve educational outcomes for high-need young children by enhancing the quality of preschool programs; (b) invitational priority 4, which invites applications that propose to expand statewide longitudinal data systems to include or integrate data from early childhood programs, among other programs; and (c) invitational priority 5, inviting applications in which the State plans to address how early childhood programs, K-12 schools, postsecondary institutions, workforce development organizations, and other State agencies and community partners, will coordinate to improve all parts of the education system and create a more seamless pre- kindergarten-20 route for students. While the Secretary is interested in applications that meet these invitational priorities, we decline to require that statewide longitudinal data systems include additional information about early childhood programs because that would go beyond the data elements specified in the America COMPETES Act. Changes: None. Timeline Comment: Many commenters suggested that a State be evaluated based on the degree of progress it has made on developing a system that would comply with the America COMPETES Act rather than on the extent to which a State has completed these efforts. Another commenter suggested a State be judged on a plan to implement any missing elements of its statewide longitudinal data system. Several commenters also stated that it is not feasible for some States to have a completed statewide longitudinal data system to be in place by September 30, 2011, the date specified in the notice of proposed requirements for the State Fiscal Stabilization Fund. Discussion: The State Reform Conditions Criteria are used to assess a State's past progress and its success in creating conditions for reform in special areas related to the four ARRA education reform areas. A State will be judged on the extent to which it has, already in place, a statewide longitudinal data system that includes the elements in the America COMPETES Act. Some commenters misunderstood criterion (C)(1); this notice does not require the statewide longitudinal data system to be completed by a particular date. Rather, a State will receive points for the elements it has completed at the time it submits its application. Changes: None. [[Page 59742]] Development of a Statewide Longitudinal Data System Comment: Several commenters stressed the importance of stakeholder support and technical expertise in the development and implementation of statewide longitudinal data systems. Some commenters suggested that we provide incentives to encourage States to design data systems using input from professional standards boards. Other commenters recommended seeking feedback from parents, businesses, educators, community-based partners, universities, hospitals, and students on the content and overall effectiveness of the statewide longitudinal data system. Discussion: We agree with commenters that stakeholder and expert support in developing a longitudinal data system is important. However, we believe that each State is in the best position to determine how best to solicit technical expertise and stakeholder support and from which groups. Accordingly, we do not believe it is necessary to specify the input and support each State should seek. Changes: None. Comment: Some commenters suggested particular qualities of strong statewide longitudinal data systems. They argued that data sets must be common across districts, cross-operational, and supportive of developing a robust, accurate, and immediately useful data mine. Commenters emphasized the importance of developing data systems that are comprehensive, systemic, reliable, valid, and designed for long- term use. One commenter suggested that the Department ensure data elements are used to create uniform cohorts. Discussion: We agree with commenters that these are important characteristics of a statewide longitudinal data system. We believe that the 12 data elements in the America COMPETES Act represent the qualities suggested by the commenters, and therefore, no change is necessary. Changes: None. Comment: One commenter recommended that the State data systems should reflect sufficient grade-to-grade alignment in order to ensure that valid grade-level growth determinations can be made in each State. This commenter urged that the Department require that such growth measures be used only with vertically scaled assessments that are appropriate for examining value-added growth. Two commenters recommended emphasizing the importance of States using cohort data in the statewide longitudinal data systems for determining student progress. Discussion: We agree with commenters who emphasize the importance of data and assessment systems that support the measurement of student growth. In this notice, student growth is defined as the change in achievement data for an individual student between two or more points in time. A State may also include other measures that are rigorous and comparable across classrooms. Given this definition, we decline to specify or restrict the structure of statewide longitudinal data or assessment systems but rather allow States the flexibility to develop data and assessment systems, as long as they support a growth measure that is rigorous and comparable across classrooms. Changes: None. Comment: Many commenters stressed that it was important for States to develop interoperable data systems that are aligned with existing technology platforms and able to incorporate data from existing data management systems. Commenters also stressed the importance of ensuring that statewide longitudinal data systems can ``communicate'' with each other so that the data in these systems can be used by early childhood centers and institutions of higher education, within and among schools, within and among LEAs, among State and local agencies, across States and with Federal agencies. One commenter requested that the Department provide additional clarification regarding America COMPETES Act element (4), ``the capacity to communicate with higher education data systems'' and whether this capacity includes data integration or two-way communication. Discussion: The COMPETES Act requires a statewide longitudinal data system to have the capacity to communicate with higher education data systems. Therefore, statewide longitudinal data systems should have the ability to link an individual student record from one system to another, consistent with FERPA, including 34 CFR Part 99. Additionally, these systems should meet interoperability and portability standards, which will ensure that they have timely and reliable opportunities to share data across different sectors within a State and across States. Timely and reliable information from across sectors will facilitate the evaluation of which program or combinations of programs is improving outcomes for students. Note that States must consider how to protect student privacy as data are shared across agencies. Successful applicants that receive Race to the Top grant awards will need to comply with FERPA, including 34 CFR Part 99, as well as State and local requirements regarding privacy. Changes: None. Selection Criterion (C)(2) (proposed Selection Criterion (B)(2)): Accessing and using State data: Uses of Data Comment: Several commenters expressed support for our proposal to evaluate State Race to the Top applications based on the extent to which the State plans to use this data to inform and engage key stakeholders, such as policymakers, parents, students, and the public, so that they have information about how well students are performing. Many commenters recommended that these data systems should also be used to identify continuous improvement goals, address barriers that compromise student success, and highlight understanding of best practices. Some commenters suggested these data systems be used to improve instructional practice by facilitating the use of differentiated instruction, to make individualized decisions about students' academic and developmental needs, and to design comprehensive interventions to address those needs. A few commenters suggested that States use these data systems to inform professional development and teacher and administrator evaluations, evaluate teacher preparation programs, allow for the monitoring of teacher and principal assignments, and ensure equitable distribution of teachers. One commenter suggested that data be used to address conditions that lead to the racial isolation of low income students. Commenters recommended that data systems be used to inform strategic planning, inform resource allocation decisions, and support decision-makers in overall organizational effectiveness. In order to ensure that all students have equitable access to education, one commenter recommended that data be analyzed to identify and implement an appropriate array of options that use early access to college coursework as a way to promote college readiness for every student. Discussion: Criterion (C)(2) will be used to evaluate a State on the extent to which it has a high-quality plan to ensure that the data from its statewide longitudinal data system are accessible to, and used to inform and engage decision-makers in the continuous improvement of policy, instruction, operations, management, resource allocation, and overall effectiveness. We [[Page 59743]] agree with the commenters that data from these systems can be used for many of the purposes indentified by the commenters. However, we believe most of these are covered in the broad categories of instruction, operations management, and resource allocation. We are revising the criterion to specify that such data can also be used in the areas of ``policy'' and ``overall effectiveness.'' Changes: We have revised criterion (C)(2) to include ``policy,'' and ``overall effectiveness'' as areas for which data may be used. Building Capacity Comment: Several commenters stated that the Race to the Top funds should be used to build State capacity for data accuracy, analysis, and dissemination. One commenter urged the Department to consider ways to help States expand and use longitudinal data systems. Other commenters recommended that a State be judged on its capacity to use the data contained in these systems or how it has moved from collecting data to transforming the data into actionable information for use. Discussion: We agree with the commenter that State plans under this criterion should include a proposal for how the State will improve its own capacity to analyze and use data. We believe the criterion makes this clear and that no further changes are needed. In addition, invitational priority 4 indicates that the Secretary is particularly interested in applications in which States propose working together to adapt one State's statewide longitudinal data system so that it may be used, in whole or in part, by other States, rather than having each State build or continue building such systems independently. We will consider the commenter's request for the Department to help States expand their statewide longitudinal data systems as we develop plans to provide technical assistance to grantees. Changes: None. Accessibility of Data Comment: Some commenters recommended adding language to criterion (C)(2) to ensure that data from a State's statewide longitudinal data system are accessible to key stakeholders. For instance, commenters suggested requiring a State to describe how its data are presented in a format and language that key stakeholders can access and understand, and are in a format that is easy to interpret and analyze. One commenter suggested that this notice compel a State to describe the format (e.g., dashboards, reports, data downloads) and timelines in which it plans to provide the appropriate level of data to the different stakeholders, as well as its communication plans to ensure that stakeholders are aware this information is available. Some commenters were especially concerned that the data are accessible to communities and families, and in particular, that these stakeholders be provided support in understanding data and their uses to monitor children's progress and to hold districts and schools accountable. A few commenters recommended that States and LEAs provide parents and the public with clear and concise annual reports that are useful and relevant to all constituencies. Commenters suggested topics that should be included in these reports, such as an overall assessment of education, reports on school quality, descriptions of progress in the core academic subjects, and indicators of the health and safety of children. One commenter suggested that States include in reports an opportunity-to-learn index to track data about the quality of State and local education systems. Another commenter suggested that reports provide teachers with data on the growth of their students on interim or summative assessments. A few commenters noted the importance of consultation with stakeholders after the data are reported, recommending that States and LEAs address in their application how they plan to disseminate and explain the data to stakeholders and how they will use community input to develop a plan of action to improve schools. Discussion: We agree with commenters that data should be accessible to key stakeholders and that reports including those data should provide useful information to them. A State's application will be evaluated on the extent to which it has a high-quality plan to make sure its data are accessible to, and used to inform and engage key stakeholders. However, we decline to specify the exact format of the data, what might be included in reports, the specific input or consultation with stakeholders, or the timelines for sharing data given the unique nature of statewide longitudinal data systems and the differing needs of constituencies within States. These are all potential elements that States could include, however, in their Race to the Top plans. Changes: None. Comment: Some commenters suggested adding to the list of stakeholders in criterion (C)(2) other groups who should have access to data from statewide longitudinal data systems, such as families (instead of parents), youth-serving community-based organizations and value-added intermediaries, parent teacher associations, nonprofit organizations, workforce investment boards, business leaders, community groups, institutions of higher education involved in the preparation of new teachers, and early childhood program providers. Discussion: The list of stakeholders in criterion (C)(2) is meant to be illustrative, but not exhaustive. States should make data available, consistent with FERPA, including 34 CFR Part 99, to any relevant stakeholder it deems appropriate. We do not, however, think it is necessary to add more examples of stakeholders to this criterion. Changes: None. Comment: Many commenters recommended that the Department require a State to address how public charter schools will have the same access to the information produced by these data systems as traditional public schools. Commenters believed that access to high-quality student-level data is critical to the successful operation of all public schools, including public charter schools, and is a key underpinning of any accountability based system. Another commenter requested that the Department clarify that charter schools must provide data to States. Discussion: The Department agrees that charter schools should have the same access to the information produced by statewide longitudinal data systems as traditional public schools and States should ensure this access. Nothing in this notice would prohibit equal access to data for public charter schools. Public charter schools must provide States with any data specified by the State on the same basis as other public schools. Changes: None. Privacy Issues Comments: Several commenters recommended that the Department require a State to provide assurances concerning the safeguards it has in place to protect the privacy of students and school employees as data about them are shared. Discussion: States must consider how to protect student privacy as data are shared. Successful applicants that receive Race to the Top grant awards will need to comply with FERPA, and its implementing regulations 34 CFR Part 99, as well as any applicable State and local requirements. Because a State's compliance with FERPA is a requirement with which all recipients of Department funds must meet, we are removing the reference to compliance [[Page 59744]] with FERPA from the text of the selection criteria in (C). To remind States of their obligations under FERPA, we are including a footnote with a reference to the statute and implementing regulations in this section. The Department agrees that teacher and principal privacy also must be protected. However, teacher and principal privacy is governed by State law. States, LEAs, and schools should consider their individual State privacy statutes when addressing these privacy issues in the establishment of a statewide longitudinal data system. Changes: We moved the references to FERPA from the criteria in (C) to a footnote in that same section. Comments: Several commenters stated that the Department should harmonize Federal policy to ensure that individual privacy protections are safeguarded in a way that does not interfere with timely and necessary information sharing. Some commenters expressed concern that States may face challenges in fully implementing statewide longitudinal data systems while meeting the requirements of FERPA unless current FERPA regulations regarding data-sharing among State agencies are revised. They recommended that the FERPA regulations be revised to explicitly allow for interagency data exchanges so the Administration's policy goals for Race to the Top can be realized. Discussion: The Department recognizes that further clarity on FERPA and the America COMPETES Act will facilitate States' ability to develop and implement statewide longitudinal data systems that contain all 12 data elements outlined in the America COMPETES Act. The establishment of a statewide longitudinal data system with the necessary functionality to incorporate all 12 of the COMPETES Act elements, by itself, does not violate FERPA. The actual implementation of such a system (including the disclosure and redisclosure of personally identifiable information from education records) also does not violate FERPA provided that States follow FERPA's specific requirements. For example, the Department's current interpretation of FERPA is not a barrier to importing data into an educational agency from another State agency, since FERPA only applies to the personally identifiable information contained in education records. In the following discussions, in response to specific questions from commenters, we provide greater detail about how a statewide longitudinal data system may be established and implemented in compliance with FERPA. The Department is not aware of any other Federal laws that would prohibit or pose barriers to a State establishing a statewide longitudinal data system. To the extent that State laws present barriers to the development of a statewide longitudinal data system in compliance with the ARRA, the State will likely need to take specific actions to address those barriers. The Department will provide further clarification in this area as warranted. Changes: None. Comments: Some commenters asked the Department to clearly explain how post-secondary institutions, K-12, and pre-kindergarten-K education systems can share restricted student information. Discussion: As stated previously, the establishment of a statewide longitudinal data system with the necessary functionality to incorporate all 12 of the COMPETES Act elements, including the sharing of data between pre-kindergarten-12 and postsecondary data systems, by itself, does not violate FERPA. States also may implement a statewide longitudinal data system that includes the disclosure and redisclosure of personally identifiable information from education records in a manner that complies with FERPA. In addition to complying with FERPA, any sharing of student data must also comply with the requirements of 34 CFR 104.42(b)(4) (the regulations implementing section 504 of the Rehabilitation Act), generally prohibiting postsecondary institutions from making pre-admission inquiries about an applicant's disability status. We first address the question of the disclosure and redisclosure of personally identifiable information in the pre-kindergarten context. The disclosure of personally identifiable information from pre- kindergarten to LEAs is not affected by FERPA with respect to pre- kindergarten programs that do not receive funding from the Department, as FERPA does not apply to those programs. With respect to pre- kindergarten programs that receive funding from the Department, the non-consensual disclosure of personally identifiable information from the students' pre-kindergarten education records to LEAs is permitted under the enrollment exception in the FERPA regulations, provided that certain notification and access requirements are met. (20 U.S.C. 1232g(b)(1)(B); 34 CFR 99.31(a)(2) and 99.34). The second issue raised by commenters involved the sharing of information between postsecondary institutions and SEAs. Similar to the pre-kindergarten context, the non-consensual disclosure of personally identifiable information from K-12 education records to a postsecondary institution is permitted under the enrollment exception, provided the notification and access conditions are met. Postsecondary institutions may disclose personally identifiable information to an SEA under the evaluation exception if the SEA has the authority to conduct an audit or evaluation of the postsecondary institution's education programs. (20 U.S.C. 1232g(b)(1)(C), (b)(3), and (b)(5); 34 CFR 99.31(a)(3) and 99.35). States that have not established the requisite authority may do so in a number of ways, such as: (1) Creating an entity in the State to house the statewide longitudinal data system and endowing that entity with the authority to conduct evaluations of elementary, secondary, and postsecondary education programs; or (2) granting authority at the SEA or IHE level to conduct evaluations of elementary, secondary and postsecondary education programs. States may grant authority through various vehicles, including for example, Executive Orders, regulations and legislation. In some States the formation documents for SEAs, IHEs or other educational entities may already grant the necessary authority; however, explicit statutory authority is not required by FERPA. The Department recognizes that there is considerable variation among States' governance structures and laws, and that using this exception to obtain personally identifiable information from postsecondary institutions may be difficult. The Department is currently reviewing its regulations and policies in this area and will be in close communications with States over the next several months regarding these issues. Of course, the Department also is available, upon request, to provide States with technical assistance on how to implement a statewide longitudinal data system that meets the requirements of FERPA. Changes: None. Comment: A few commenters requested that the Department provide specific guidance about the de-identification process that all States must adhere to in order to share potentially identifiable information about students. Discussion: It is not possible to prescribe or identify a single method to minimize the risk of disclosing personally identifiable information in redacted records or statistical information that will apply in every circumstance, including determining whether defining a minimum cell size is an appropriate means to protect the [[Page 59745]] confidentiality of aggregated data and, if so, selection of an appropriate number. This is because determining whether a particular set of methods for de-identifying data and limiting disclosure risk is adequate cannot be made without examining the underlying data sets, other data that have been released, publicly available directories, and other data that are linked or can be linked to the information in question. For these reasons, we are unable to provide examples of rules and policies that necessarily meet the de-identification requirements in 34 CFR 99.31(b). The releasing party is responsible for conducting its own analysis and identifying the best methods to protect the confidentiality of information from education records it chooses to release. We recommend that State educational authorities, educational agencies and institutions, and other parties refer to the examples and methods described in the notice of proposed rulemaking to amend its FERPA regulations that the Department published in the Federal Register on March 24, 2008 (73 FR 15574, 15584) (FERPA notice of proposed rulemaking) and refer to the Federal Committee on Statistical Methodology's Statistical Policy Working Paper 22, http://www.fcsm.gov/working-papers/spwp22.html, for additional guidance. Further, as noted in the preceding paragraph and in the preamble to the FERPA NPRM, use of minimum cell sizes or data suppression is only one of several ways in which information from education records may be de-identified before release. Statistical Policy Working Paper 22 describes other disclosure limitation methods, such as ``top coding'' and ``data swapping,'' which may be more suitable than simple data suppression for releasing the maximum amount of information to the public without breaching confidentiality requirements. Decisions regarding whether to use data suppression or some other method or combination of methods to avoid disclosing personally identifiable information in statistical information must be made on a case-by-case basis. With regard to issues with ESEA reporting in particular, determining the minimum cell size to ensure statistical reliability of information is a completely different analysis than that used to determine the appropriate minimum cell size to ensure confidentiality. Changes: None. Selection Criterion (C)(3): Using data to improve instruction (proposed Selection Criterion (B)(3)): Comment: One commenter recommended that a State describe in its plan the State and LEA roles and responsibilities related to using data to improve instruction, including how the plan would ensure that LEAs are primarily responsible for creating instructional improvement systems with assistance and support from the State. One commenter recommended that the Department increase the explicit emphasis on adoption and implementation of local data and instructional improvement systems. Discussion: Application requirement (e)(4) requires States to describe, for each Reform Plan Criteria that it chooses to address, the parties responsible for implementing the activities. We therefore do not feel it is necessary to specify in the criterion itself that a State should describe its roles and responsibilities and that of its LEAs. However, we agree with the commenters that criterion (C)(3)(i) concerns local instructional improvement systems, and we are revising it to clarify this. We are also clarifying that the plans under this criterion should include efforts to increase the acquisition and adoption of such systems. Changes: Criterion (C)(3)(i) now begins, ``Increase the acquisition, adoption, and use of local instructional improvement systems.'' Comment: Some commenters suggested that a State be evaluated on the degree to which it can demonstrate collaboration and cooperation with and among LEAs. Several commenters recommended that the Department include an incentive for States and LEAs to learn from outstanding LEAs in data development and reporting in order to improve vertical alignment of the State's education system. Discussion: As described elsewhere in this notice, States receiving Race to the Top funds, along with their LEAs and schools, are expected to identify and share promising practices, make work freely available to others and make data available in appropriate ways that comply with FERPA to stakeholders and researchers. Specifically, criterion (A)(1)(ii) provides for the evaluation of a State based on the extent to which the participating LEAs are strongly committed to the State's plans and to effective implementation of reform in the four education areas. Criterion (A)(2)(i)(b) asks the State to demonstrate how it will support participating LEAs in successfully implementing the education reform plans the State has proposed, through such activities as identifying promising practices, evaluating these practices' effectiveness, ceasing ineffective practices, widely disseminating and replicating the effective practices statewide, holding participating LEAs accountable for progress and performance, and intervening where necessary. In addition, under criterion (C)(3)(i), a State will be evaluated on the extent to which it, in collaboration with its participating LEAs, has a high-quality plan to increase the LEAs' acquisition, adoption, and use of local instructional improvement systems that provide teachers, principals, and administrators with the information and resources they need to improve their instructional practices, decision-making, and overall effectiveness. This could include facilitating collaboration between LEAs. Given these existing criteria, we do not believe a change is necessary. Changes: None. Comment: One commenter suggested that the Department allow States to focus on early childhood care and development data systems exclusively, without penalty for not including K-12 instructional improvement systems. Discussion: While we believe it is important for instructional improvement systems to include tools for improving early childhood care, we decline to make the commenter's suggested change. Section 14005(c) of the ARRA requires a State, when applying for a Race to the Top grant, to describe the status of the State's progress in each of the four assurance areas in section 14005(d), including improving the collection and use of data. We believe the assurance in the ARRA related to the use of data is intended to cover all levels of the educational system. Changes: None. Comment: Several commenters recommended revising criterion (C)(3)(i) to include other stakeholders, in addition to teachers and principals, who can benefit from using data to improve instruction, such as youth development professionals in after-school and summer programs, mentoring and after-school learning organizations, expanded learning time partners, early childhood providers, and program directors. Discussion: We understand that there are other stakeholders outside of the school who play critical roles in education. Criterion (C)(2) addresses how data from a statewide longitudinal data system can be used by a wide range of stakeholders, whereas criterion (C)(3)(i) is focused on how data are specifically used in instructional improvement systems to improve instructional practices, decision-making, and overall effectiveness during the school day. We believe the list of stakeholders in criterion (C)(3)(i) is appropriate given this focus, therefore, we do not believe it is necessary to [[Page 59746]] revise this criterion. However, nothing in this notice would prevent a State from specifying in its plan additional stakeholders who may use instructional improvement systems. Changes: None. Comment: Some commenters stated that, in addition to making data available, there must also be an equal focus on building the capacity of educators and school leaders to analyze and use this information. They argued that a State should describe how it will support its LEAs in providing effective, collaboratively designed and research-based professional development, including pre-service training to teachers, principals and administrators on how to analyze and use these data. One commenter suggested that professional development opportunities include a focus on using multiple sources of information to assess student academic performance; using a variety of strategies to analyze data; using data to identify barriers for success, design strategies for improvement, and plan daily instruction; benchmarking successful schools with similar demographics to identify strategies for improvement; and, creating a school environment that makes data-driven decisions. One commenter suggested that a State should articulate the means by which it will require educators seeking certification or re- certification to receive training and show competence in the analysis, interpretation, and use of data. Several commenters suggested that time during the school day should be dedicated to data analysis and action planning for teachers. Another commenter suggested that a State be required to explain how it will promote an environment (e.g., a climate of autonomy) in which teachers, principals, and administrators have the support and conditions to make decisions based on the results of the data analyses. Discussion: We agree with commenters that States must support their LEAs in providing effective professional development. We are adding a new criterion (C)(3)(ii) to encourage States to support participating LEAs and schools that are using local instructional improvement systems to provide effective professional development to teachers, principals, and administrators on how to use these systems and the resulting data to support continuous instructional improvement. We are also clarifying, in criterion (C)(3)(i), that the purpose of instructional improvement systems is to provide educators with the resources they need, as well as the information they need. In addition, criterion (D)(5) addresses the need for high-quality professional development. The Department also encourages States to utilize current Federal education funding, for example Title II-A Improving Teacher Quality State grants, as a funding mechanism to provide further professional development to teachers in the use of data in the classroom. We do not believe we should require a State to articulate the means by which it will require educators seeking certification or re- certification to receive training and show competence in the analysis, interpretation, and use of data. A State may address this issue in its plan if it chooses. Changes: Criterion (C)(3)(ii) has been added to provide that a State will be evaluated based on the extent to which it has a high- quality plan to support LEAs and schools that are using instructional improvement systems (as defined in this notice) in providing effective professional development to teachers, principals, and administrators on how to use these systems and the resulting data to support continuous instructional improvement. As a result of this addition, proposed criterion (C)(3)(ii) has been redesignated (C)(3)(iii). We have also revised criterion (C)(3)(i) to clarify that instructional improvement systems should provide educators with the ``information and resources they need to inform and improve their instructional practices, decision-making, and overall effectiveness.'' Comment: One commenter did not support making data available and accessible to researchers. This commenter stated that large urban districts are deluged with requests for information and access to data, which diverts time and resources from student-centered activities, and that this misconstrues the purpose of Race to the Top to improve student achievement and close achievement gaps. Rather than making data available to researchers for the purposes specified in criterion (C)(3)(iii), this commenter suggested that the data be available instead to evaluation contractors and State and Federal officials. Discussion: We appreciate the commenter's concern about the resources needed to share data with researchers. However, we believe it is very important that researchers, consistent with FERPA, including 34 CFR Part 99, be able to conduct studies to improve instruction. We therefore decline to make the recommended change to make the data available only to evaluation contractors and State and Federal officials. Changes: None. Comment: Many commenters suggested that the Department clarify that instructional improvement systems should identify students who are off- track to graduation or have dropped out of school. These commenters said that early warning indicators can be used by LEAs and States to develop and implement options that will keep students on track, or put them back on track, to graduation. Discussion: We agree with the commenter that instructional improvement systems should provide early warning indicators about students at risk of educational failure and are revising the definition of instructional improvement systems accordingly. We also are revising criterion (C)(3)(iii) to be consistent with criterion (C)(3)(ii) and to clarify that the data from instructional improvement systems, together with statewide longitudinal data system data, should be made available and accessible to researchers. Changes: We have revised the definition of instructional improvement systems to clarify that such systems may also integrate instructional data with other student-level data such as attendance, discipline, grades, credit accumulation, and student survey results to provide early warning indicators of a student's risk of educational failure. We have also revised criterion (C)(3)(iii) to clarify that the data from ``instructional improvement systems,'' together with statewide longitudinal data system data, should be made available and accessible to researchers. Comment: One commenter recommended that the Department clarify the definition of instructional improvement systems to reference use of technology-based tools and other strategies to systemically manage cycles of continuous instructional improvement. A few commenters suggested that instructional improvement systems should be research- based. Some commenters suggested that the definition of this term should state that the purposes of these systems are to: Ensure that every student has access to instructional materials that are current and aligned to these standards; differentiate instruction; provide individualized learning; gather input and feedback from stakeholders; translate data into knowledge; drive innovation; use knowledge to create networks of best practices; and inform decision-making. Discussion: In response to these comments, we are clarifying the [[Page 59747]] definition of instructional improvement systems. However, we are not specifying additional purposes of instructional improvement systems, as this could inadvertently discourage States and LEAs from developing new and innovative strategies for addressing students' learning needs. In response to the commenters who indicated that instructional improvement systems should be research-based, we believe that much research has been done on the effectiveness of using data to inform instructional decisions. Instructional improvement systems provide teachers and instructional leaders with the evidence they need to make informed instructional decisions. Therefore, such systems are a critical element of any classroom-based, evidence-driven approach to instruction. Changes: We have revised the definition of instructional improvement systems to reference that such systems are ``technology- based tools and other strategies that provide teachers, principals, and administrators with meaningful support and actionable data to systemically manage continuous instructional improvement * * *.'' In addition, we have included summative assessments as an additional example of information gathering on instructional improvement. Performance Measures and Minimum Evidence for Selection Criteria (C)(1), (C)(2), and (C)(3) Comment: Several commenters recommended specific performance measures for criteria (C)(1), (C)(2), and (C)(3). For instance, one commenter recommended that data performance measures include indices or rankings on districts' and schools' actual provision of basic resources and opportunities that the ARRA contemplates. Another commenter encouraged the Department to include a performance measure that States must ensure data are in a format and in a language that families can access and understand, consistent with the myriad roles parents are required to play under the ESEA. Another commenter recommended that performance measures for criterion (C)(2) include the results of surveys of stakeholders. One commenter suggested that performance measures be used to evaluate the extent to which the output from the statewide longitudinal data system is geared to stakeholder needs. Discussion: A State may propose its own performance measure(s) for the section on Data Systems to Support Instruction. Rather than requiring particular performance measures for this section, we are choosing to give a State the flexibility to define its own measures that are tailored to the context of its statewide longitudinal data system. Changes: None. Comment: One commenter suggested that criterion (C)(3)(iii) require minimum evidence to ensure that competing applications are judged in a consistent manner. Another commenter recommended that minimum evidence should include the adoption and publication of procedures for the request and release of longitudinal data for research purposes. In addition, this commenter suggested that evidence include the State's partnerships with national researchers to evaluate the effectiveness of the instructional practices in each participating LEA. Discussion: We believe that the basic elements of a plan, as specified in Application Requirement (e), should be sufficient to yield consistent judging on this criterion. We therefore decline to require the specific minimum evidence suggested by the commenters. D. Great Teachers and Leaders Selection Criterion (D)(1): Providing High-Quality Pathways for Aspiring Teachers and Principals (Proposed Criterion (C)(1)): Comment: Many commenters recommended changes to the proposed definition of alternative certification routes. Two commenters suggested changing the term to ``alternative routes to certification'' to be consistent with the terminology in criterion (D)(1). Some commenters recommended that the definition refer to school districts and nonprofit organizations as providers of programs offering alternative routes to certification. A few commenters sought to ensure that programs offering alternative routes to certification be selective in accepting candidates into their programs. Many commenters objected to defining an alternative route to certification as one that includes clinical or student teaching experience, claiming that such experiences are characteristic of traditional preparation programs, and that other kinds of training, such as intensive mentoring support during the first months of teaching, are more valuable than clinical or student teaching experiences. However, one commenter supported field-based experiences for principals, and other commenters stated that administrators seeking alternative routes to certification should have prior teaching experience. Commenters also had different views on the level and type of coursework that should be part of alternative routes to certification. One commenter supported alternative routes to certification involving limited amounts of coursework, one commenter disagreed, and a third commenter specifically recommended requiring substantive coursework in reading and math content and teaching methods. Several commenters recommended that the definition include a requirement that all alternative routes to certification ensure that graduates of such programs have the skills to address the needs of all students. One commenter expressed concern that alternative routes to certification, given their shortened timeframe, are not designed to ensure that teachers develop the skills needed to effectively instruct students with disabilities. The commenter recommended strengthening both traditional and alternative route preparation programs so that all teachers are more skilled in teaching students with disabilities. Two commenters sought changes aimed at ensuring that graduates of alternative routes to certification receive the same level of certification as teachers and leaders who complete traditional preparation programs. Similarly, a few commenters recommended that the Department require States to verify that teachers certified through alternative routes to certification are treated equally and fairly in hiring under all State regulations and statutes, while another commenter suggested sanctioning States that treat alternative routes to certification as a ``route of last resort.'' On the other hand, one commenter stated that teachers certified through alternative routes generally should not be assigned to high-need schools because of their limited experience. Discussion: In response to these comments, the Department is making a number of changes to the definition of alternative certification routes. First, we agree that the various terms used in the Race to the Top program should be consistent; therefore, we are changing the proposed term ``alternative certification routes'' to ``alternative routes to certification'' in this notice. We also agree that the NPP was unclear regarding providers of alternative routes to certification, and are changing the definition to clarify that qualified providers of States' teacher and administrator preparation programs include both institutions of higher education and other providers that operate independently from institutions of higher education. In addition, we agree that providers of alternative routes to certification, as with all preparation programs, should be selective in enrolling individuals in their programs [[Page 59748]] and, therefore, are changing the definition to ensure that qualified providers of teacher and principal preparation programs are selective in the candidates they accept. The Department believes it is important to provide prospective teachers and principals with direct school and classroom experiences as part of their preparation. Because alternative routes to certification are accelerated and vary in delivery models, there are a variety of ways, in addition to clinical or student teaching experiences, to provide this experience, such as through practicum and job embedded experiences, coupled with intensive mentoring or support during the first months of teaching, as suggested by the commenters. We agree with the commenters and are revising the definition to refer to school-based experiences and ongoing support such as effective mentoring and coaching. As to the extent of the coursework required by programs providing alternative routes to certification, the Department believes that States are in the best position to determine the courses and coursework that could be reduced or limited as a part of any alternative route to certification program, consistent with the needs of local schools, the accelerated nature of alternative routes to certification, and the wide range of previous education and experience that candidates bring to these programs. The Department, therefore, declines to change the definition to specify the amount or type of coursework that must be included in programs providing alternative routes to certification. We are specifying in the final definition, however, that alternative routes to certification should include standard features such as demonstration of subject-matter mastery and high-quality instruction in pedagogy. We also believe that programs providing alternative route to certification should not award levels of certification that are different from the certifications available from traditional preparation programs, which could limit the opportunities for teachers to teach and leaders to lead; rather, alternative routes to certification programs, whether for teachers or principals, should be considered different pathways to certification with the same rigor as other State-approved routes. The Department's view is that States, LEAs, and schools should treat individuals prepared through State- approved alternative routes to certification in the same manner as those prepared and certified through traditional teacher and principal preparation programs, and we are changing the definition to reflect this view. The Department agrees that there is a need to strengthen preparation programs to prepare teachers and principals to meet the needs of all students. We are revising the definition of alternative routes to certification to clarify that such routes should prepare teachers and principals to address the needs of all students, including English language learners and students with disabilities. Changes: We have changed the term ``alternative certification routes'' to ``alternative routes to certification.'' We also have made the following changes: (1) Revised clause (a) to clarify that ``other providers'' refers to ``other providers operating independently from institutions of higher education''; (2) added a new clause (b) to clarify that alternative routes to certification programs must be selective in accepting candidates; (3) re-designated proposed clause (b) as new clause (c) and changed ``clinical/student teaching experiences'' to ``supervised, school-based experiences and ongoing support such as effective mentoring and coaching;'' (4) re-designated proposed clause (c) as new clause (d); and (5) re-designated proposed clause (d) as new clause (e) and revised it to clarify that upon completion, programs providing alternative routes to certification must award the same level of certification that traditional preparation programs award upon completion. We have also revised the definition of alternative routes to certification to clarify that such routes should include ``standard features such as demonstration of subject-matter mastery, and high-quality instruction in pedagogy and in addressing the needs of all students in the classroom including English language learners and students with disabilities.'' Comment: Many commenters suggested that the Race to the Top competition places too much emphasis on alternative routes to certification and recommended that the Department eliminate the focus on alternative routes and expand the criterion to include multiple routes. Several commenters expressed concern that alternative routes to certification are not as effective as traditional routes. Those commenters argued that alternative routes to certification do not provide the necessary skill sets to impact teaching and learning, and do not attract educators with the necessary background to provide instructional leadership. A few commenters questioned whether criterion (D)(1) is necessary. One commenter recommended that the Department not require States to require alternative routes for principals. A few commenters argued that research shows that alternative routes have not been as effective as traditional programs. One commenter suggested that the Department focus on the quality of pathways to certification rather than the number of those pathways. Multiple commenters suggested that States develop common standards of performance for those entering the profession, regardless of the route taken. One commenter recommended that the Department establish safeguards to ensure that alternative routes successfully prepare candidates to meet a consistent set of standards that govern teacher licensure. A few generally supportive commenters recommended monitoring these routes to ensure quality programs, and requiring States to provide evidence of a quality control process for their certification programs. Discussion: The Department agrees that we should encourage the creation of high-quality pathways for aspiring teachers and principals through both traditional and alternative routes to certification. We are therefore adding criterion (D)(1)(iii), under which States will be rewarded for having a process for monitoring, evaluating, and identifying areas of teacher and principal shortage and for preparing teachers and principals to fill these areas of shortage. At the same time, we believe it is important to retain the original substance of proposed criterion (C)(1), regarding alternative routes to certification, for two reasons. First, to increase the supply of high- quality talent entering the field of education we must reduce the barriers to entry into the education profession, especially for high- achieving individuals, such as individuals who have changed careers and recent college graduates who have the potential to be good educators. Alternative routes to certification are typically optimized for such new entrants into the profession. Second, the Secretary believes that competition between traditional and alternative certification providers will help increase the quality of all programs. To provide clarity, and to emphasize the importance of alternative routes actually being in use, we are separating proposed criterion (C)(1) into two criteria, (D)(1)(i) and (D)(1)(ii). To further support the notion that all teacher and administrator preparation programs must train candidates to become high-performing professionals, we proposed in the NPP and establish in this final notice, criterion (D)(4). This [[Page 59749]] criterion is intended to shine a light on the quality of all preparation programs in the State by providing both potential candidates and schools recruiting graduates with valuable information about which programs are actually best preparing candidates for success. We are also adding criterion (D)(4)(ii), which encourages States to expand preparation and credentialing options and programs that are successful at producing effective teachers and principals. Together, we believe that criteria (D)(1) and (D)(4) provide a combination of rewards, incentives, and transparency that could result in significant quality improvements in educator preparation and recruitment. Finally, we do not believe we should remove principals from this criterion. Well-prepared principals are critical to providing the instructional leadership necessary to support teaching and learning in our schools. We know that chronically underperforming schools too often have poor leadership, and that poor leadership drives away good teachers. The focus on principal preparation is therefore critical. Changes: Criterion (D)(1) now reads, ``Providing high-quality pathways for aspiring teachers and principals: The extent to which the State has-- (i) Legal, statutory, or regulatory provisions that allow alternative routes to certification (as defined in this notice) for teachers and principals, particularly routes that allow for providers in addition to institutions of higher education; (ii) Alternative routes to certification (as defined in this notice) that are in use; and (iii) A process for monitoring, evaluating, and identifying areas of teacher and principal shortage and for preparing teachers and principals to fill these areas of shortage.'' In addition, we have added criterion (D)(4)(ii), which encourages States to ``expand preparation and credentialing options and programs that are successful at producing effective teachers and principals (both as defined in this notice).'' Comment: One commenter recommended including an additional requirement that States demonstrate the extent to which their alternative routes for STEM teachers draw upon nationally recognized models. Discussion: The Department places great emphasis in Race to the Top on STEM, as evidenced by the fact that we have established a competitive preference priority for STEM proposals in this notice. We also recognize the importance of using models that have shown success in raising student achievement in STEM areas. However, we do not believe it is necessary to require that States demonstrate the extent to which their alternative routes to certification for STEM teachers utilize nationally recognized models. We expect that all alternative routes to certification, including those for STEM teachers, would include standard features such as demonstration of subject-matter mastery, and high-quality instruction in pedagogy and in addressing the needs of all students in the classroom including English language learners and student with disabilities. As previously stated, we are adding language to the definition of alternative routes to certification that clarifies this point. Changes: None. Comment: Two commenters recommended that a portion of the Race to the Top funds be used to promote new approaches to alternative routes to certification, incentivizing existing programs to adopt research- based and effective strategies. Discussion: The Department recognizes that there are many research- based, innovative practices that can help teachers, principals, and others improve student achievement. Nothing in this notice prevents States from engaging in or supporting such innovation. The Department notes that it recently announced proposed priorities, requirements, definitions, and selection criteria for the Investing in Innovation Fund. Established under section 14007 of the ARRA, the Investing in Innovation fund will provide competitive grants to expand the implementation of innovative practices that show the promise of significantly improving K-12 student achievement for high-need students, as well as help close the achievement gap, and improve teacher and principal effectiveness. The grants will allow eligible entities to expand their work and serve as models of best practices. LEAs and nonprofit organizations interested in developing new approaches to improve teacher and principal effectiveness in meeting the needs of high-need students and scaling-up such strategies may wish to consider applying for an Investing in Innovation grant. Changes: None. Comment: One commenter recommended that, instead of asking States to show the extent to which they encourage alternative routes to certification, States should be required to demonstrate the extent to which teacher preparation programs partner with high-need LEAs and schools to meet the specific personnel needs of those LEAs and schools. Discussion: The Department agrees that creating partnerships between effective teacher preparation programs and high-need LEAs and schools could be an effective strategy to meet personnel needs. As discussed earlier, we are adding criterion (D)(1)(iii), which is focused on identifying areas of teacher and principal shortage and preparing teachers and principals to fill them. States could address part of this criterion by establishing the partnerships suggested by the commenter. Changes: None. Selection Criterion (D)(2): Improving Teacher and Principal Effectiveness Based on Performance (Proposed Criterion (C)(2)): Comment: Several commenters recommended requiring that teacher and principal evaluations be conducted at the local level and that States only provide support rather than be directly involved in the evaluation process. Many commenters also stated that the consequences of those evaluations (e.g., performance pay) should also be decided at a local level. Those commenters argued that local school systems are better able to identify effective and ineffective educators, allowing for meaningful comparisons and interpretations across schools. Another commenter recommended adding an assurance encouraging States to provide local control to principals over issues such as hiring, leadership team appointments, school-based funding, and scheduling flexibility. Two commenters suggested replacing ``differentiating'' in the title of criterion (D)(2) (proposed criterion (C)(2)) with ``evaluating.'' Other commenters stated that the focus of this criterion should be primarily on improving the performance of teachers and principals in order to improve student achievement. Discussion: It was the Department's intent that LEAs would be the entities conducting teacher and principal evaluations and making informed decisions, based on the evaluations, regarding teacher and principal development, compensation, promotion, retention, tenure, and removal. We are revising criterion (D)(2) to clarify that participating LEAs (as defined in this notice) should perform these functions and States should have a plan for ensuring that participating LEAs do so. While differentiating performance is an important component of evaluation systems, we agree that criterion (D)(2) is focused on improving teacher and principal effectiveness, and we are changing the title to make this clear. We [[Page 59750]] also have made the development of evaluation systems (rather than differentiation) the centerpiece of this criterion by revising (D)(2) to encourage the design and implementation of high-quality evaluation systems, and to promote their use for feedback, professional improvement, and decision-making. Changes: We have revised criterion (D)(2) to clarify that the State's role is to ``ensure that participating LEAs'' perform the functions described in criterion (D)(2). We have also replaced ``differentiating'' with ``improving'' in the title of criterion (D)(2). We have also reframed this criterion so that it focuses on the creation and use of evaluation systems. Comment: One commenter recommended changing criterion (D)(2)(i) (proposed criterion (C)(2)(a)) to read ``Establish and provide a clear description of a system to measure impact on student growth (as defined in this notice) that uses a rigorous statistical approach.'' Discussion: We accept the commenter's suggested language, in part. We do not, however, believe it is necessary to include in criterion (D)(2)(i) that the measure of student growth uses a rigorous statistical approach. The definition of student growth in this notice already provides that the approaches used to measure growth must be rigorous. We are changing criterion (D)(2)(i) to reflect the first part of the commenter's suggested language. We are also clarifying that growth should be measured for each individual student. Changes: Criterion (D)(2)(i) has been revised to read, ``Establish clear approaches to measuring student growth (as defined in this notice) and measure it for each individual student.'' Comment: One commenter asked for clarification regarding the word ``overall'' in the proposed definition of an effective principal. Discussion: The word ``overall'' in the definition of effective principal refers to the performance of all of the students in the school, taken as a whole. The analogue from the ESEA is the ``all students'' group used in AYP determinations. We are removing the reference to section 1111(b)(2)(C)(v)(II) of the ESEA from the definition of effective principal because, as noted elsewhere, a new paragraph (g) in the Application Requirements section of this notice explains that references to ESEA subgroups throughout the notice are the subgroups described in section 1111(b)(2)(C)(v)(II) of the ESEA. Changes: We have removed the parenthetical ``(described in section 1111(b)(2)(C)(v)(II) of the ESEA)'' from the definition of effective principal. Comment: Many commenters stated that the proposed definition of effective principal relies too heavily on standardized test scores as the sole measure of effectiveness. Several commenters recommended that the definition be changed to require States to expand the definition beyond student growth to include multiple measures such as effectiveness as a leader; effective fiscal management; student, community, and parental engagement; effective school safety; evidence of providing a supportive teaching and learning environment; discipline; college matriculation rates; college readiness rates; and data on staff turnover rates and working conditions. One commenter suggested balancing the evaluation of principals by including data from State assessments and other data on student learning in all core academic subjects, so as to avoid ``narrowing the curriculum.'' Other commenters emphasized the principal's role in creating a positive school climate, engaging students, increasing the number of effective teachers, continuous improvement, connecting learning to solving community problems, implementing school-wide practices that drive substantial student achievement gains, and preparing students for success in work and post-secondary education. One commenter suggested supplementing the definition to state that an effective principal is one who demonstrates growth in the number and percentage of effective and highly effective teachers within the school through demonstrated success in strategies such as teacher recruitment and selection, retention, high quality data-driven professional development, feedback and coaching to individual teachers, counseling out, and fair dismissals. Discussion: The Department believes that student growth must be a significant factor in determining principal effectiveness. However, we agree with commenters that data on student growth should not be used as the sole means of evaluating principals and that States, LEAs, and schools should supplement student growth with other measures of effectiveness. Accordingly, we are revising the definition of effective principal to require that they do so. While we cannot include in the definition all of the measures recommended by the commenters, we believe it is important to include several examples for illustrative purposes and are adding examples of the following measures in the definition of effective principal: high school graduation rates and college enrollment rates, as well as evidence of providing supportive teaching and learning conditions, strong instructional leadership, and positive family and community engagement. We also are making minor changes to the definition for purposes of clarification. Changes: We have changed the definition of effective principal as follows: (a) Replaced ``States may supplement this definition as they see fit'' with ``States, LEAs, or schools must include multiple measures;'' (b) added ''Supplemental measures may include, for example, high school graduation rates and college enrollment rates, as well as evidence of providing supportive teaching and learning conditions, strong instructional leadership, and positive family and community engagement;'' and (c) replaced ``so long as principal effectiveness is judged, in significant measure by student growth'' with ``provided that principal effectiveness is evaluated, in significant part, by student growth.'' Comment: One commenter supported the definition of effective teacher and agreed that student growth should be used as a measure of teacher effectiveness along with other supplemental measures. However, many commenters stated that the proposed definition relies too heavily on standardized test scores and recommended requiring supplemental measures. Another commenter recommended giving States the flexibility to define effective teachers using models that make sense in their States. Several commenters suggested that the definition include examples of supplemental measures such as using research-based teaching practices, implementing practices that have been documented in the classrooms of teachers who are driving substantial student achievement gains, and using feedback and student performance data to improve teaching. Discussion: As noted in our response to commenters' concerns that student growth data should not be used as the sole means to evaluate principals, we agree with commenters that States, LEAs, and schools should include multiple measures in determining teacher effectiveness. We are, therefore, changing the definition to require States, LEAs, or schools to take into account data on student growth as a significant measure of teacher effectiveness, but also to include multiple measures. We also are adding multiple observation-based assessments of teacher performance as an example of [[Page 59751]] a supplemental measure in the definition of effective teacher. Changes: We have defined effective teacher to mean ``a teacher whose students achieve acceptable rates (e.g., at least one grade level in an academic year) of student growth (as defined in this notice). States, LEAs, or schools must include multiple measures, provided that teacher effectiveness is evaluated, in significant part, by student growth (as defined in this notice). Supplemental measures may include, for example, multiple observation-based assessments of teacher performance.'' Comment: One commenter recommended that the definition of effective teacher be changed to require student growth to be a ``predominant measure,'' rather than a ``significant measure,'' of teacher effectiveness. The commenter noted that using student growth as a ``significant measure'' for judging teacher effectiveness would allow other factors to outweigh a teacher's impact on student achievement. Discussion: We believe that having student growth as a significant factor in determining teacher effectiveness is a sufficiently rigorous standard. The revised definition also provides States, LEAs, and schools with more flexibility in determining the appropriate use of supplemental measures without outweighing the importance of teachers' impact on student growth in determining teacher effectiveness. Changes: None. Comment: Several commenters suggested that the definition of effective teacher acknowledge and address the need to mentor and support new teachers who disproportionately work in struggling schools. Discussion: We agree that professional development, including mentoring and coaching, are important aspects of teacher effectiveness. For this reason, criterion (D)(2)(iv)(a) focuses on using evaluations to inform decisions regarding developing effective teachers and principals, including by providing relevant coaching, induction support, and/or professional development. Criterion (D)(5) also provides for evaluation of the extent to which a State has a high- quality plan for its participating LEAs to provide effective, data- informed professional development, coaching, induction, and common planning and collaboration time to teachers and principals. We believe these criteria address the need for mentoring and other forms of professional development for teachers and therefore, are not changing the definition of effective teacher in the manner recommended by the commenter. Changes: None. Comment: One commenter strongly recommended including high school graduation rates as a measure to evaluate teacher effectiveness in order to provide a disincentive to ``creaming'' students and to signal the importance of preventing students from dropping out. Discussion: We believe it could be misleading to include high school graduation rates as a required or supplemental measure of teacher effectiveness, because, more than other measures, graduation rates typically reflect the work of many teachers and school administrators. Accordingly, we have included graduation rates as an example of a supplemental measure of effectiveness in the definitions of effective principal and highly effective principal. Changes: None. Comment: One commenter recommended that effective teacher be defined as a teacher whose students, overall and for each subgroup, demonstrate acceptable rates of student growth. The commenter noted that the definition of effective principal refers to ``each subgroup'' and expressed concern that the omission of ``each subgroup'' in the definition of effective teacher could be misinterpreted to mean that teachers could be deemed effective (or highly effective) even if their students from different subgroups are not making sufficient learning gains. Discussion: The Department included the performance of subgroups in the definitions of effective principal and highly effective principal because there would generally be a sufficiently large number of students in a particular subgroup at the school level to evaluate principal effectiveness. However, it is generally unlikely that a class would have a sufficient number of students in any particular subgroup on which to base an evaluation of a teacher's effectiveness. Changes: None. Comment: Some commenters recommended that, instead of defining effective teacher, this notice should encourage the use of proven tactics for improving teacher effectiveness (e.g., lowering class sizes or innovative solutions for addressing the challenges teachers face). Other commenters suggested encouraging States to develop and use performance assessments of teachers that reliably and validly assess the use of teaching practices known to be associated with student achievement gains and to experiment with a range of strategies to incorporate evidence of student learning and accomplishment into teacher evaluation tools. One commenter recommended that educators should use research data and scientific recommendation as a basis for instruction and developing appropriate methods. Discussion: Throughout this final notice, the Department encourages States, LEAs, and schools to use proven strategies for improving teacher effectiveness and addressing other challenges teachers face. For example, Invitational Priority 6--School-Level Conditions for Reform, Innovation, and Learning focuses on providing schools with flexibility and autonomy, such as creating school climates and cultures that remove obstacles to, and actively support, student engagement and achievement, and implementing strategies to effectively engage families and communities in supporting the academic success of their students. Criterion (C)(3) focuses on using data to improve instruction by increasing the acquisition, adoption, and use of local instructional improvement systems that provide teachers and principals with the information they need to inform and improve instructional practices; supporting LEAs and schools that use these systems in providing professional development on how to use these systems to support instructional improvement; and making data available and accessible to researchers so they can evaluate the effectiveness of instructional materials, strategies, and approaches. Criteria (D)(2)(iv)(a) and (D)(5) emphasize that the supports provided to teachers and principals should be ongoing and informed by data and evaluations. Changes: None. Comment: One commenter expressed concern that data on student growth are available only for the limited number of subjects included in the annual assessments required under the ESEA. The commenter recommended that we clarify that alternative measures of student performance should be used for teachers teaching subjects that are not tested under the ESEA. Another commenter asked how teacher effectiveness would be determined when there are no data on student growth, such as might be the case for novice teachers and teachers teaching subjects or grades that are not tested under the ESEA. Discussion: As defined in this notice, the term student growth means the change in student achievement (as defined in this notice) for an individual student between two or more points in time. In turn, the definition of student achievement includes alternative measures of student performance for [[Page 59752]] non-tested grades and subjects. As noted elsewhere, we are adding, in the definition of student achievement, a number of examples of alternative measures of student performance for both tested and non- tested grades and subjects and clarifying that for tested grades and subjects, student achievement must include a student's score on the State assessments required under the ESEA (which will allow for the determination of student growth) and may include other measures of student learning as well. Therefore, we do not believe that additional language needs to be added to the definition of effective teacher. Changes: None. Comment: One commenter expressed concern that the definition of effective teacher equates effectiveness with advancing students one grade level in an academic year. The commenter stated that this approach ignores the fact that research has not identified a standard for student gains in a given school year in a given subject. Another commenter requested clarification regarding the meaning of ``at least one grade level in an academic year'' as used in the definition of effective teacher. Another commenter inquired whether States that use summative tests to measure one or more years of student growth would need to change their assessment system. Discussion: We included ``at least one grade level in an academic year'' as an example of an acceptable rate of student growth in the definition of effective teacher (and effective principal). We recognized that this example of an acceptable rate of student growth may not be appropriate for all students and therefore, did not include it as a requirement but rather as an example. We believe States, LEAs, and schools should determine what constitutes an acceptable rate of student growth for purposes of assessing teacher (or principal) effectiveness. Changes: None. Comment: As with the definition of effective principal, many commenters expressed concern about using student growth as the sole measure for defining a highly effective principal. Some commenters stated that a good measure of a highly effective principal is success in attracting, developing, and retaining effective teachers. Another commenter, however, stated that significant growth in student achievement would suffice as evidence of a highly effective principal's ability to improve teacher effectiveness. Discussion: As noted earlier, the Secretary believes that student growth must be included as a significant factor in evaluating principal and teacher effectiveness. However, he understands and appreciates commenters' concerns that student growth should not be used as the sole means to evaluate principals and teachers. Therefore, we are changing the definition of highly effective principal, consistent with the changes to the definition of effective principal, to require States, LEAs, or schools to take into account multiple measures, in addition to data on student growth, in defining a highly effective principal. We agree with commenters that success in attracting, developing, and retaining high numbers of effective teachers would be a good measure of a highly effective principal and are adding this to the definition along with other examples of supplemental measures. We also are making minor technical changes for clarity and removing the statutory reference to section 1111(b)(2)(C)(v)(II) of the ESEA, regarding student subgroups. We are removing the statutory reference to the ESEA because, as noted elsewhere, a new paragraph (g) in the Application Requirements section of this notice explains that references to ESEA subgroups throughout the notice are the subgroups described in section 1111(b)(2)(C)(v)(II) of the ESEA. Changes: We have changed the definition of highly effective principal to read as follows: ``Highly effective principal means a principal whose students, overall and for each subgroup, achieve high rates (e.g., one and one-half grade levels in an academic year) of student growth (as defined in this notice). States, LEAs, or schools must include multiple measures, provided that principal effectiveness is evaluated, in significant part, by student growth (as defined in this notice). Supplemental measures may include, for example, high school graduation rates; college enrollment rates; evidence of providing supportive teaching and learning conditions, strong instructional leadership, and positive family and community engagement; or evidence of attracting, developing, and retaining high numbers of effective teachers.'' Comment: One commenter noted that the definition of highly effective principal refers to ``high rates of student growth'' and recommended modifying the definition of student growth accordingly. Discussion: We believe that States' definition of highly effective principal should demonstrate high rates of student growth for their students overall, and for each subgroup. The Department believes that one and one-half grade levels of growth in an academic year is a good example of a high rate of student growth. We recognize, however, that this example of ``high rates of student growth'' may not be appropriate for all students. We included ``one and one-half grade levels in an academic year'' as an example, not a requirement. We believe States, LEAs, and schools should determine what constitutes a high rate of student growth, as the definitions of highly effective principal (and highly effective teacher) clearly permit. We, therefore, do not believe it is necessary to revise the definition of student growth, as requested by the commenter. Changes: None. Comment: One commenter suggested that successful completion of a State-approved principal licensure program that builds the knowledge, skills, and attitudes to effectively lead people, lead learning, and manage school operations should be included as a measure of a highly effective principal. Discussion: States, LEAs, and schools may choose to use successful completion of a State-approved principal licensure program as a supplemental measure of a highly effective principal. However, we decline to include it as an example of a supplemental measure in the definition of a highly effective principal because we believe that principal effectiveness is best determined by measuring results and outcomes. Changes: None. Comment: Some commenters commended the Department for focusing the definition of teacher effectiveness on student achievement and growth. Other commenters recommended adding language that would allow States and LEAs to supplement student growth with multiple measures determined on the State or local level. Other commenters suggested that States and LEAs be required to supplement their definitions of student growth with multiple measures. Commenters also recommended that such measures include the use of evidence-based practices for improving student achievement, the use of feedback and professional development opportunities, and leadership activities such as mentoring or leading an instructional community. One commenter did not believe the definition should include a teacher's commitment and ability to use feedback and performance data to improve instructional practices. The commenter reasoned that a teacher who improves student achievement is using (1) practices that are both effective for student learning and healthy for social and emotional development of students and (2) feedback to improve practice. One commenter urged the Department [[Page 59753]] to have ``an equity focus on those current highly qualified teacher proxies that have some research base grounded in student achievement: Novice and out of field teaching.'' Another commenter suggested that the definition provide individual school districts with the flexibility to establish policies to determine whether a teacher is highly effective in order to ``recognize that a wide range of conditions can vary from district to district that would make a state-wide definition inappropriate for evaluation, promotion, or compensation purposes.'' Discussion: We agree with commenters that States, LEAs, and schools should be required to supplement their definition of a highly effective teacher with multiple measures. We are, therefore, revising the definition to require that States, LEAs, or schools include multiple measures. In addition, we are including examples of supplemental measures that States, LEAs, and schools might use, including leadership roles. Changes: We have revised the definition of highly effective teacher to mean a teacher whose students achieve high rates (e.g., one and one- half grade levels in an academic year) of student growth (as defined in this notice). States, LEAs, or schools must include multiple measures, provided that teacher effectiveness is evaluated, in significant part, by student growth (as defined in this notice). Supplemental measures may include, for example, multiple observation-based assessments of teacher performance or evidence of leadership roles (which may include mentoring or leading professional learning communities) that increase the effectiveness of other teachers in the school or LEA. Comment: One commenter suggested that adopting the definitions of effective teacher and highly effective teacher in the NPP would be at odds with the value-added system prescribed in the State Fiscal Stabilization Fund. Discussion: The definitions of effective teacher and highly effective teacher in this notice are not at odds with the requirements of the State Fiscal Stabilization Fund. The Race to the Top definitions are broad enough to give States, LEAs, and schools sufficient flexibility to determine the approach to measuring growth that works best for them, giving them a variety of ways to comply with the requirements of the State Fiscal Stabilization Fund. Changes: None. Comment: One commenter suggested that standardized tests are not created to measure teacher effectiveness and therefore are an invalid measure of effectiveness. Discussion: We believe students' standardized test scores are one of many measures that can be used to determine student growth. However, we recognize that teacher effectiveness should not be determined solely on the basis of standardized test scores, which is why we are requiring, in this final notice, the use of student growth as a significant factor in teacher evaluations that must include multiple measures. Changes: None. Comment: Commenters stressed that it is imperative that there is common ground on how to develop, fairly compensate, and accurately evaluate teachers. A few commenters stated that there should be collaboration between teachers and principals in determining appropriate measures for evaluation. Discussion: We agree about the importance of involving teachers and principals in the design and development of these evaluation systems, and are adding in this final notice language requiring such systems to be designed and developed with teacher and principal involvement. Changes: We have revised criterion (D)(2)(ii) to read, ``Design and implement rigorous, transparent, and fair evaluation systems for teachers and principals that (a) differentiate effectiveness using multiple rating categories that take into account data on student growth (as defined in this notice) as a significant factor, and (b) are designed and developed with teacher and principal involvement.'' Comment: One commenter recommended that the Department replace the word ``rating'' with ``personnel evaluation'' to account for a more nuanced approach with multiple measures. Discussion: We believe that the reference to ``rating categories'' in criterion (D)(2)(ii) is sufficiently clear that the criterion does not need to be revised. Changes: None. Comment: A large number of commenters recommended changes to the proposed definition of student growth. Some suggested that we include in the definition the use of non-achievement-based measures of student learning, performance-based or portfolio assessments, and interim assessments. Other commenters suggested including in the definition the specific amount of growth required. Some commenters supported the proposed definition's emphasis on individual growth, while others called for comparisons among ``like populations,'' such as students with disabilities or English language learners. One commenter warned that the use of a growth-based model could make teachers unwilling to serve students with disabilities. Some commenters urged the Department to require specific models, such as value-added, while others urged the Department not to require specific models in order to leave States with the flexibility to develop their own measures of student growth. One commenter was concerned that the definition ``amounts to another all or nothing model'' and pointed out that research on student growth cautions against making judgments about student growth using solely two data points, and suggested that we reconsider this approach. Discussion: Our purpose, in the context of a competitive grant program intended to provide leading-edge States with incentives to develop and test innovative education reform ideas, is to give States freedom to create their own systems for measuring student growth within a few key parameters. We believe that the proposed definition strikes this balance and that, therefore, significant changes are not needed. We acknowledge that LEAs or schools may reasonably want to measure student growth using more than two data points. We are changing the phrase ``two points in time'' to ``two or more points in time'' to permit the use of interim assessments or achievement data collected across multiple years. We are also editing the second sentence for clarity; this includes deleting the phrase ``in order to increase the construct validity and generalizability of the information.'' Changes: We have revised the definition of student growth to read as follows: ``Student growth means the change in student achievement (as defined in this notice) for an individual student between two or more points in time. A State may also include other measures that are rigorous and comparable across classrooms.'' Comment: Numerous commenters expressed their support for evaluating teachers and principals based on student achievement or growth. One commenter stated that principal evaluations should include an aggregation of data on student growth. Several of these commenters, however, asserted that student growth data have limitations, including a lack of common definitions between States, difficulty in disaggregating a teacher's effect on student achievement from other effects, and the lack of data for all grade levels and subject areas. Additionally, many commenters expressed their disapproval of the proposed criteria regarding using student achievement data or student [[Page 59754]] growth for the evaluation of teachers and principals. In support of their arguments, those commenters cited factors such as the current limitations of student assessments, and the inadequacy of assessments as an evaluation factor. Several of those commenters claimed that there is a lack of research or evidence demonstrating that the use of such data for teacher and principal evaluations has any positive impact on teacher, principal, or student performance. One commenter disagreed with the Department's statement that ``It is difficult to predict teacher quality based on the qualifications that teachers bring to the job. Indeed measures such as certification, master's degrees, and years of teaching experience have limited predictive power on this point.'' The commenter argued that the research the Department cites (i.e., Kane et al.) actually demonstrates that teaching experience and whether a teacher is fully certified does indeed have substantial impact on students' achievement. Other commenters argued that research indicates growth models are unstable and too vulnerable to multiple sources of error and to other student and school factors separate and apart from student achievement. Additionally, many commenters offered reasons for not using student assessments as a factor in teacher and principal evaluations, including the claims that: Using student achievement data to make employment decisions may lead to corruption, students are not held accountable for the results of State assessments, and that such a policy would detract from other priorities, such as equitable distribution of effective teachers. Another commenter argued that measuring teacher effectiveness ignores the organizational context of schools and inappropriately defaults to a single measure of student test scores as the basis to evaluate, compensate, and dismiss teachers. Discussion: Research shows that teacher quality is a critical contributor to student learning, and that differences between teachers are persistent. Kane et al. found in their study that the certification status of teachers (e.g., certified, uncertified, and alternative certified) ``has at most small impacts on student test performance.'' At the same time, they found that, ``among those with the same certification status, there are large and persistent differences in teacher effectiveness.'' They also reported that evidence suggests that teachers' classroom performance during their first two years of teaching is a more reliable indicator of a teacher's future effectiveness than their certification status.\4\ Another study used data from Chicago public high schools to estimate the importance of teachers on student achievement in mathematics and found that, ``one semester with a teacher rated two standard deviations higher in quality could add 0.3 to 0.5 grade equivalents, or 25 to 45 percent of an average school year, to a student's math score performance.'' The study further concluded that the resulting teacher quality ratings ``remain relatively stable for an individual instructor over time.'' \5\ A recent study of New York City public charter schools concluded that charter schools that pay teachers in part based on evaluations of their performance have more positive effects on student achievement.\6\ In light of this evidence, the Department believes that the best indicator we have today for teacher (and by extension principal) quality is student academic growth, but that (as noted above) this data must be supplemented with additional measures. At the same time, the Secretary appreciates that growth models are not yet perfect, that there are some challenges to using student growth data, and that there is more work to be done in this area. For this reason, we do not stipulate which approach States, LEAs, or schools should use to measure student growth so long as the approach used is rigorous and comparable across classrooms (see the definition of student growth). The criteria and definitions in this notice reflect the Department's belief that student growth data should be used as a significant factor in determining teacher and principal effectiveness; that evaluation systems should use multiple measures; that these evaluation systems should be rigorous, transparent, and fair; and that they should be designed and developed with teacher and principal involvement. --------------------------------------------------------------------------- \4\ See, Kane, Thomas J., Jonah E. Rockoff, and Douglas O. Staiger (2006), ``What Does Certification Tell Us About Teacher Effectiveness? Evidence from New York City.'' \5\ Daniel Aaronson, Lisa Barrow, and William Sander (2003), ``Teacher and Student Achievement in the Chicago Public High Schools,'' Federal Reserve Bank of Chicago Working Paper 2002-28. \6\ Hoxby, Caroline M., Sonali Murarka, and Jenny Kang. ``How New York City's Charter Schools Affect Achievement, August 2009 Report.'' Second report in series. Cambridge, MA: New York City Charter Schools Evaluation Project, September 2009. --------------------------------------------------------------------------- We do not agree that using student growth data as a part of a rigorous, transparent, and fair evaluation system that is designed and developed with teacher and principal involvement will lead to corruption or detract from other priorities. We contend that implementing fair and transparent evaluation systems developed with the involvement of both teachers and principals, and that include student growth as a significant factor in evaluations, will lead to greater trust between teachers and principals, enable meaningful decision- making and support, and push educators to remain focused on the ultimate priority -- improving student achievement. Changes: None. Comment: One commenter recommended that the Department encourage the development of research-based rubrics and/or innovative teacher performance evaluation programs. Discussion: We encourage LEAs to be innovative and draw on rigorous research in creating evaluation systems; this is an area that has high leverage and is ripe for change. However, in order to avoid creating a one-size-fits-all policy or stifling innovation, we decline to name specific tools that LEAs should use in their evaluation systems. Changes: None. Comment: One commenter recommended that the Department consider designating NAEP as the standard test for every State to measure student achievement. Discussion: Race to the Top will use both the NAEP and the assessments required under the ESEA to measure student achievement. Each test has its benefits and its drawbacks; together, we believe they will offer the Nation an appropriate ``picture'' of how Race to the Top States are performing. Changes: None. Comment: One commenter recommended removing the phrase ``targeted professional development'' from criterion (D)(2)(iv)(a) (proposed criterion (C)(2)(d)(i)). The commenter's rationale was that the Department should promote a comprehensive system for managing and developing human capital rather than a one-to-one system based on remediation. In addition, the commenter asserted that the Department should be explicit that professional development must be for the purpose of increasing student achievement. Discussion: We agree with the commenter that the term ``targeted professional development'' does not connote the appropriately broad range of professional development and support for teachers and principals originally envisioned by the Department. We are therefore changing this criterion to include the phrase ``providing relevant coaching, induction support, and/or professional development.'' We do, however, want to make clear that in the context of criterion (D)(2), we are encouraging LEAs and schools to consider how they will use teachers' and principals' [[Page 59755]] evaluations to inform their specific professional development plans. In other criteria, such as (D)(5) and (C)(3)(ii), we encourage a broad range of professional development activities. We also believe that, by specifying that professional development should be responsive to evaluations that use student growth as a significant factor, we make clear in this final notice that professional development should be oriented around supporting teachers and principals in increasing student achievement. Changes: We have split proposed criterion (C)(2)(d)(i) into two parts. We have combined the first part with proposed criterion (C)(2)(c), resulting in criterion (D)(2)(iii), which reads, ``Conduct annual evaluations of teachers and principals that include timely and constructive feedback; as part of such evaluations, provide teachers and principals with data on student growth for their students, classes, and schools.'' The second part has been designated criterion (D)(2)(iv)(a), which specifies that evaluations should inform decisions regarding ``Developing teachers and principals, including by providing relevant coaching, induction support, and/or professional development.'' Comment: A few commenters recommended that the Department include a clear statement indicating that State reform plans should specify that teachers and principals will be assessed on more than a single year of data. Discussion: We believe it is important to use accurate data when evaluating teacher and principal performance, and that those evaluations should be done at least annually and should involve timely and constructive feedback. To make it clear, however, that annual evaluations do not have to be conducted based on only one year of information, we have revised the definition of student growth to clarify that student growth should be measured using achievement data between ``two or more points in time,'' rather than between only two points in time. Changes: We have revised the definition of student growth so that it means the change in achievement data for an individual student between ``two or more points in time.'' Comment: A number of commenters supported the use of student growth data in determining compensation and promotions. A few commenters stated that the Department needs to specify how to structure performance pay (e.g., how to offer it for teachers of subjects that are not tested). However, many commenters expressed their opposition to pay based on student achievement or growth data. Several commenters stated that there is no evidence suggesting that performance pay linked to achievement data leads to improved educational outcomes. Several commenters asserted that performance pay places an undue emphasis on teachers and principals as individuals as opposed to parts of the education system as a whole. One commenter recommended that Race to the Top funds be used to design tests in pilot districts that could test the effectiveness of alternative compensation programs. Discussion: The Department believes that we need to do much more to shine a spotlight on and reward excellence in teaching and school leadership, and that one way to do so is through compensation and promotion. At the same time, we recognize that rewarding excellence while fulfilling the demands of fairness and the need to maintain a collaborative school environment is a delicate task that requires cooperation between LEA leadership, principals, and teachers. We also recognize that pay-for-performance systems in education are controversial and spark much debate. Some States, LEAs, and schools have experimented with such models and shown relative success and promise, while others have experienced less encouraging results. The ARRA also includes funds for the Teacher Incentive Fund, which will award grants to LEAs to develop performance-based compensation models. While research on pay-for-performance plans is limited, there is evidence to suggest that a well-designed performance-based pay system can lead to improved student achievement.\7\ Studies indicate that the most effective and successful pay-for-performance systems incorporate factors such as using multiple measures for evaluating performance; making student growth just one measure of performance; having a clearly identified purpose (e.g.. improving student achievement, improving recruitment and retention, or attracting teachers to hard-to-staff schools); and creating collaboration among teachers, principals, and other stakeholders. The Department believes that criterion (D)(2) incorporates these factors by specifying that evaluation systems for teacher and principals should use multiple measures, take into account student growth as a significant factor, and be designed and developed with teacher and principal involvement. --------------------------------------------------------------------------- \7\ See e.g., Joshua H. Barnett, Gary W. Ritter, Marcus A. Winters, and Jay P. Greene, ``Evaluation of Year One of the Achievement Challenge Pilot Project in the Little Rock Public School District,'' University of Arkansas, January 2007. --------------------------------------------------------------------------- We also note that the criterion refers to decisions regarding promotion and retention as well as compensation because we believe that great teaching and school leadership should be recognized and rewarded as much as possible, and that talented educators should have opportunities for increased responsibilities and other retention incentives, where appropriate, as well as for additional compensation. Changes: We have reorganized criterion (D)(2) to make it clearer that the decisions discussed in criterion (D)(2)(iv) should be based on the evaluation systems discussed in criterion (D)(2)(ii) and the evaluations discussed in criterion (D)(2)(iii). We have also added ``retaining'' to the list of decisions in criterion (D)(2)(iv)(b). Comment: Numerous commenters argued that performance pay would create perverse incentives for teachers to work only with student groups most likely to demonstrate improvement, thereby marginalizing difficult-to-teach student groups and communities, including low-income communities, English language learners, and students with disabilities. Discussion: As contemplated in the notice, performance pay would be based on teacher and principal evaluations that, as discussed previously, use student growth--not raw student achievement data or proficiency levels--as a significant factor. Thus, teachers whose pupils start behind their peers or who are working with students with disabilities or English language learners are in no way penalized. This final notice also gives States, LEAs, and/or schools sufficient flexibility to take these concerns of commenters into account when creating systems for evaluation, compensation, and promotion. We also note that the Department is placing an emphasis on attracting teachers to hard-to-staff subjects, specialty areas, and schools in criterion (D)(3). Changes: None. Comment: Several commenters recommended including language requiring States to provide additional responsibilities for effective teachers. Many of the commenters included specific examples of professional opportunities States or LEAs should provide to highly effective teachers, such as serving as a community liaison, induction leader, or curriculum developer after earning an endorsement on their teacher's license. Discussion: The Department believes that it is critical to adequately compensate and promote our best [[Page 59756]] teachers and principals. These professionals are the role models and leaders of our schools and are essential to implementing effective educational reforms and improving student achievement. For these reasons, this notice makes clear that highly effective teachers and principals should have an opportunity to obtain additional compensation and responsibilities for their high performance. We believe that LEAs and schools, in collaboration with their teachers and principals, are best situated to determine the timing and types of additional responsibilities that should be given to their staff and that it would be inappropriate for the Department to set requirements around this issue. Changes: None. Comment: One commenter recommended replacing the word ``tenure'' with ``continuing employment status'' for the sake of clarity. Discussion: The Department believes the word ``tenure'' is more widely understood and declines to make the suggested change. Changes: None. Comment: Several commenters expressed concern that, while proposed criterion (C)(2)(iii) mentions using information to grant tenure and dismiss teachers, it does not focus on the need to retain teachers. One of these commenters stated that dismissals are going to involve a very small percentage of teachers and principals. The commenter further stated that both rural and urban schools may have difficulty attracting and retaining effective teachers. One commenter cited the difficulties in attracting and retaining effective or highly effective teachers in extremely rural areas. The commenter further stated that school districts in rural areas are forced to hire beginning teachers who cannot be considered effective or highly effective as defined in the NPP. A couple of commenters believed that robust, strong, and fair evaluation systems are important for attracting and retaining highly qualified, effective teachers and principals to high-poverty schools. Discussion: The Department concurs that recruiting and retaining effective and highly effective teachers and principals is critical for States and LEAs to meet their goals for education reform and improve student achievement, particularly in high-poverty and/or high-minority schools. For this reason, criterion (D)(3) discusses the equitable distribution of effective teachers and principals in high-poverty and/ or high-minority schools and encourages States and LEAs to provide incentives and strategies to attract and retain effective teachers and principals. Criteria (D)(2)(iv)(a) and (D)(5) also encourage States to support LEAs in providing professional development and undertaking other efforts, especially those informed by data and evaluations, to make their existing teachers more effective. We are also revising criterion (D)(2)(iv)(b) to specifically clarify that teacher and principal evaluations should inform retention decisions. Changes: We have revised criterion (D)(2)(iv)(b) to read as follows, ``Compensating, promoting, and retaining teachers and principals, including by providing opportunities for highly effective teachers and principals (both as defined in this notice) to obtain additional compensation and be given additional responsibilities.'' Comment: Many commenters supported using evaluations in making employment decisions, such as those regarding teacher and principal tenure, dismissal, displacement, and layoff. Most of these commenters supported using multiple measures in these evaluations and not basing such employment decisions primarily or solely on assessment results. Discussion: We agree that rigorous, transparent, and fair evaluation systems should be used to inform a variety of decisions, including development, compensation, retention, tenure, certification, and removal. As discussed earlier, we are requiring that evaluation systems include multiple measures and that student growth be a significant factor, and we are revising criterion (D)(2) to make it clearer that the decisions under criterion (D)(2)(iv) should be based on the evaluation systems discussed in criterion (D)(2)(ii) and the evaluations discussed in criterion (d)(2)(iii). For purposes of clarity, we are dividing proposed criterion (C)(2)(d)(iii) into two criteria and adding decisions regarding full certification to one of the criteria. Changes: Proposed criterion (C)(2)(d)(iii) has been reorganized as criteria (D)(2)(iv)(c) and (D)(2)(iv)(d). Criterion (D)(2)(iv)(c) addresses the use of evaluation systems to inform decisions regarding whether to grant tenure and/or full certification to teachers and principals, and criterion (D)(2)(iv)(d) addresses removing ineffective tenured and untenured teachers and principals after they have had ample opportunities to improve. For both criteria, we have clarified that these decisions should be made using rigorous standards and streamlined, transparent, and fair procedures. Comment: Some commenters suggested that the Department should clarify the statement that the removal of teachers and principals must only occur after they have received ample support and opportunities to improve their performance yet have failed to do so. A few commenters recommended that we clarify the term ``ample opportunities'' and specify the amount of time that low-performing teachers should have to improve their performance (e.g., as one school year). Discussion: Providing teachers and principals with the needed support to improve the effectiveness of instruction and student outcomes is a critical element of Race to the Top, and removing ineffective professionals from schools is important as well. Race to the Top includes a number of criteria, in addition to criterion (D)(2), that are dedicated to teacher and principal professional development and supports; parts of criteria (B)(3) and (C)(3) and all of criterion (D)(5) concern this issue, including discussions of professional collaboration and planning time, individualized development plans, training and support in the analysis and use of data, classroom observations with immediate feedback, and other activities critical to supporting and improving teacher and principal capacity. These supports are paired, in the Race to the Top criteria, with criteria that focus on rigorous, fair and transparent teacher and principal evaluation systems that should include providing feedback on areas where professional improvements are needed. We decline to specify the amount of time teachers should be given to make improvements in their performance, beyond specifying that they should have ``ample opportunities to improve.'' It is the responsibility of the LEA and school to provide their students with effective teachers and principals, to provide their teachers and principals with effective support, and to take action when appropriate. We have deleted the phrase ``but have not done so'' to reflect this. Changes: We have removed the phrase ``but have not done so'' from criterion (D)(2)(iv)(d). Comment: One commenter argued that aspects of criterion (D)(2) (proposed criterion (C)(2)) may contravene the Personnel Evaluation Standards that, according to the commenter, have been federally accredited. Discussion: The Personnel Evaluation Standards referenced by the commenter are not federally accredited or approved by the Department. They are voluntary guidelines published by a private organization and are in no way binding on the Department or its grantees. Changes: None. Comment: A few commenters recommended that States should have a [[Page 59757]] flexible amount of time to develop evaluation systems that link data on student growth to teachers and principals in order to allow time for the development of advanced assessment systems. Other commenters recommended that this notice reflect an understanding of the timeframe that may be necessary to build a comprehensive and fair teacher and principal evaluation system that takes student growth data into account given the state of the research in this area and the practical considerations in establishing such a system. The commenter stated that the proposed criterion would compel States to rush into imposing current value-added indicators of student learning on current evaluation systems rather than developing new advanced systems. Discussion: The notice does not state a specific timeframe for States to develop assessment systems and teacher and principal evaluation systems. Through their applications, States must provide, for each Reform Plan Criterion in this notice, a detailed plan for the use of grant funds that includes, among other things, (1) the key activities to be undertaken; (2) the timeline for implementing the activities; and (3) annual targets (where applicable) with respect to performance measures for the four school years beginning with the 2010- 2011 school year. (See Application Requirements, section (e), for a complete list of requirements). It is through this process that States have the flexibility to define the timeframe for implementing their activities, including systems development. States' applications will be judged, in part, on whether their activities and targets are ambitious yet achievable. As a result, we believe that this final notice appropriately encourages States and LEAs to strike the right balance between speed and thoughtfulness. We emphasize, however, that States should not wait to develop improved evaluation systems until higher- quality assessments are available, as doing so would delay this essential progress by years and, in the process, harm student achievement. We expect that these evaluation systems will improve over time, as LEAs learn from their own experiences and from the experiences of others, and as States develop higher-quality assessments, the results of which will improve the measures of student growth that feed into these evaluation systems. Changes: None. Comment: A number of commenters recommended requiring States to include in their plans a commitment to adhere to due process rights and existing State statutes concerning tenure and dismissal. A few commenters recommended requiring States to comply with local collective bargaining agreements or involve employee representatives where there is no collective bargaining agreement. One commenter specifically suggested requiring that collective bargaining be the vehicle for implementing performance pay schemes in local school districts. Discussion: In order to successfully implement many of the plans under criterion (D)(2), LEAs in collective bargaining States will need to work collaboratively with their local unions. Because this work and collaboration are so important, States will earn points based on the extent to which the local union leadership in their participating LEAs have signed the MOUs between the States and the LEAs indicating their intent to work in partnership with the LEAs in implementing the plans, including by addressing contractual issues such as local bargaining agreements. (See criterion (A)(1)). In addition, criterion (D)(2)(ii) creates incentives for LEAs to design and implement rigorous, transparent, and fair evaluation systems with teacher and principal involvement, while criterion (D)(2)(iv)(d) encourages LEAs to make decisions regarding removal using rigorous standards, and streamlined, transparent, and fair procedures. Changes: None. Selection Criterion (D)(3): Ensuring Equitable Distribution of Teachers and Principals (Proposed Criterion (C)(3)): Comment: One commenter recommended adding a definition of high- minority school and defining the term as a school in the highest quartile of schools in a State with respect to enrollment of minority students. The commenter also recommended adding a definition of low- minority school and defining the term as a school in the lo