![[Contents]](/icons/up.gif){kind=icon}
[Chapter 6: Cross-Case Analysis 3: Part 3 of 3] Assessment of Student Performance April 1997
Our interviews with teachers at the 16 participant schools reveal that the type and extent of professional development provided teachers with respect to the performance assessment has an impact upon teacher appropriation and assessment reform. For purposes of this discussion, "professional development" will be limited to training activities which are formally structured in some way and which involve individuals from outside the school working with teachers. We will leave our discussion of other teacher-centered, teacher-directed activities — such as joint planning time or regular meetings among teachers to share and discuss classroom experiences — for our examination of the impact of school organizational factors on teacher appropriation of assessments.
Teachers at the 16 schools participating in this study experienced a range of professional development activities. These activities included:
However, teachers at some schools participated in no, or virtually no, formal professional development activities related directly to the performance assessment. Exhibit 6-8 illustrates the primary vehicle of professional development experienced by teachers at each of the 16 schools and teachers' comments about the usefulness of these professional development activities.Exhibit 6-9 illustrates the coincidence of the primary form of professional development to teacher appropriation of the assessment technology.
A fairly clear pattern emerges in Exhibit 6-8, suggesting that the predominant model of professional development employed closely follows the level of initiation of the assessment. Specifically, states developing performance assessments favor train-the-trainer models of professional development (they are likely to be economical), while national efforts to support assessment reform (and education reform, in general) emphasize conferences and symposia that bring together teachers from across the country. Schools and districts that participate in these national education reform networks take advantage of the professional development activities they offer insofar as financial resources allow. Schools going it alone tap the expertise of outsiders insofar as they are able to afford their assistance.4
In the context of the discussion of professional development opportunities, it is useful to distinguish between school-initiated assessments and assessments initiated at the school level but participating in national education reform efforts. This distinction is important here because of the unique opportunities for professional development offered by the national education reform efforts. (The distinction is less useful where school-initiated assessments resemble each other regardless of their participation or non-participation in national reforms.)
In general, we have found that, though some teachers express ambivalence toward the training they received concerning the new performance assessment, teachers typically find these opportunities to be useful. If nothing more, teachers find it valuable to learn about the purposes, format, and logistics of state- and district-initiated assessments when they are first being introduced. Beyond issues of merely knowing what's coming, however, teachers in many schools found that the structured professional development activities they participated in helped them appropriate assessments for their own classroom use. Teachers reported improved understanding of how to construct tasks, write and use rubrics, and think about evidence of student learning.
The extent to which each of these models of professional development has contributed to teachers' abilities to use assessments and to their subsequent appropriation of the assessments is explored below. To reiterate, two effects of professional development opportunities contribute specifically to teacher appropriation of new assessments:
Train-the-Trainer Models of Professional Development
The train-the-trainer approach to professional development is used by four states and one district implementing performance-based assessment systems (Arizona, Kentucky, Maryland, Vermont, and Prince William County, Virginia). This model is the traditional model of professional development employed by state departments of education when they introduce reforms. Its advantage is that it is economical, while its disadvantage is that it does not necessarily achieve the far-reaching professional development and communication intended.
In each place where this model is being used, the format of professional development is similar. The state (or, in the case of Prince William County, Virginia, the district) conducts a training session for representatives from schools and districts throughout the jurisdiction. Participants are then charged with sharing what they have learned with other teachers in their schools and districts. Below, examples of instances in which this model has contributed little to teacher appropriation of an assessment technology and instances in which it has contributed more toward appropriation are considered.
Limited Impact on Teacher Appropriation
Teachers at three schools participating in this study suggested that this model may not have the desired effect at the school level. For example, only some teachers at Manzanita High School in Arizona remembered the training they received from their district's representative, and those who did remember it found the training to be useful only with respect to the logistics and mechanics of ASAP: they learned what would be expected of them in terms of administering the assessment, and seeing sample assessments helped them to understand the types of skills they should review with their students prior to the assessment. These teachers said that the professional development did not contribute to their understanding of performance-based assessment, for their district already engaged in extensive performance assessment which was, in these teachers opinions, more valid and useful for their teaching than the ASAP. Thus, any contribution toward Manzanita's teachers' appropriation of the ASAP performance assessment is negligible.
Teachers at Westgate Middle School in Prince William County had inconsistent recollections of their participation in any development activities with respect to the district's Applications Assessments. Teachers who participated in the district's development sessions said they were valuable opportunities for communication between teachers and district administrators, but most teachers who did not participate in the district-run sessions did not remember how they learned about what would be required of them with respect to the assessment. Similarly, teachers at Walters Middle School were unable to recollect which assessment training activities they had participated in had focused on Maryland's performance assessment and which had focused on the district's performance assessment.
In each of these three schools, the scope of professional development opportunities was quite limited in practice, though not necessarily in design: teachers, at least according to those who recalled the professional development activities at all, participated in single training sessions. Below, two more extensive professional development systems illustrate positive impacts on teacher appropriation of assessment technologies.
Positive Impact on Teacher Appropriation
In some assessment systems the train-the-trainer model involves more than just a chain of development sessions. In Kentucky and Vermont, teachers designated to serve as the school or district representative assume a role requiring ongoing work and leadership in the assessment reform effort. These teachers have titles such as "network leaders" (Vermont) and "cluster leaders" (Kentucky). Teachers who serve in this capacity are responsible not only for sharing professional development with other teachers in their schools or districts but also for facilitating administration of the assessments and scoring activities.5
Teachers at Maple Leaf Middle School testify to the usefulness of the professional development activities they participate in with respect to Vermont's portfolio assessment. The state has developed its professional development system with an eye toward building local level capacity to support the reform effort. Maple Leaf teachers attend three one-day "Network Sessions" each year, professional development activities run by the district's Network Leaders. These sessions focus on issues such as the use of benchmarks and anchor papers; sessions have also focused on scoring calibration. Teachers at Maple Leaf have expressed their increasing confidence in developing tasks appropriate for inclusion in students' portfolios and their increasing ability to glean and use the information from the portfolio system of assessment.
Changes Over Time in Teacher Response to a Professional Development System: Breckenridge Middle School
Teachers at Breckenridge Middle School in Kentucky testify to how the professional development opportunities supporting KIRIS have improved over time, thereby improving their own ability to appropriate the state's assessments — and the portfolio assessment in particular. Though these teachers said that the training they received with respect to scoring portfolio assessments was inadequate in 1993-94, they also suggested that it improved between that year and the next: as a result of improved training, they were better prepared to administer and score the portfolio part of KIRIS in 1994-95 than in 1993-94. The school's three Cluster Leaders also all stated that they are confident in their ability to teach their colleagues how to use and score the assessment. Furthermore, teachers identified a wide range of KIRIS-related topics in which they had received training during the 1994-95 school year, including how to include students with disabilities in KIRIS, how to generate open-ended writing prompts, how to encourage writing across the curriculum, and how to integrate portfolios into daily classroom instruction. Finally, teachers at Breckenridge commented that they are now better able to work regularly with writing prompts and portfolios in their classrooms, suggesting increased appropriation of the state's performance assessment system.
Conferences, Symposia, and Institutes
Conferences, symposia, and the like are the favored model of professional development offered by the three national assessment reform efforts included in this study (the New Standards Project, the Coalition of Essential Schools, and the College Board's Pacesetter Program). In addition, these methods of providing professional development have also been adopted by one state-level assessment system (Oregon) and one district-level system (HSD2).
National Education Reform Networks: Impact of Professional Development Opportunities
Perhaps the most unique feature of the three national level reform efforts considered in this study is the opportunity they provide participating teachers to attend conferences, symposia, and institutes attended by teachers from all over the country who are also participating in the nationwide programs. These professional development opportunities share some features with the train-the-trainer activities, in that both types of professional development involve training over several days provided by outsiders to only a subset of a school's staff. However, comparing remarks about the train-the-trainer opportunities experienced by some teachers with those of teachers who participated in a national reform effort's professional development activities suggests that the latter tend to emphasize "capacity building" — expanding teachers' abilities to works with new assessment techniques — and involve "hands-on" practice with assessment tasks and rubrics, while train-the-trainer activities (with the exception of Vermont) tend more to emphasize the communication of information.
Teachers at Noakes and Ann Chester elementary schools who have participated in the New Standards Projects national conferences described some of the skills they developed at these events. For instance, teachers at these schools attended conferences to learn about benchmarking performance tasks in reading, how to write commentaries for student work, how to use portfolios, and how to develop new performance tasks and write scoring rubrics. Teachers suggested that these experiences were very valuable in helping them learn how to use performance-based assessments, including portfolio assessments, in their classrooms, and their comments give testimony to their appropriation of the types of performance-based assessment activities the New Standards Project is trying to promote.
Similarly, teachers at Sommerville High School participating in the College Board's Pacesetter Program have attended multiple Pacesetter "institutes," both during the summer and in the middle of the school year. Though these teachers suggested that the institutes were not always as accommodating of differences across participating schools as they might be, they still said that they have found the experiences (and these teachers have now attended two summer institutes and two mid-year institutes over a period of two years) to be extremely beneficial to them, enabling them to better implement Pacesetter's integrated program of curriculum, instruction, and assessment in their classrooms.
By way of contrast, the professional development opportunities available through the Coalition of Essential Schools have only contributed peripherally to teachers' ability to develop and use performance assessments at Cooper Middle School. Though Cooper teachers do say that their participation in CES has helped them think through and plan education reform at their school, they, quite simply, have not been able to afford to attend CES conferences frequently enough to apply what they may learn to their assessment practices.
Harrison School District 2's "Academy" Model of Professional Development: Impact on Teacher Appropriation of Assessment Technologies
The case of the district-developed Colorado Springs' "Assessment Academy" is an interesting and impressive effort on the part of a school district to make extensive professional development available to all teachers. The district's Academy has trained all district teachers in the district's "performance-based curriculum" in literacy; three development sessions per grade level had been conducted as of Spring 1994. The district operates the Academy drawing mostly upon the expertise of its administrators, and also by inviting outside experts to work with teachers. Teachers who have participated in capacity- and skill-building conferences and the like have, in general, responded favorably to them, reporting that these activities have improved their abilities to develop and work with performance-based assessments. According to one teacher, "The district has done a very good job of allowing teachers access to adequate support, such as training and materials;" another teacher concurred, saying, "I haven't had to go outside the district to get the support I need" to meet the assessment requirements of the performance-based curriculum.
Collaboration with Outside Experts and Other Networks
Some local level assessment reform efforts draw upon the expertise of outside experts to provide professional development for teachers. For instance, in South Brunswick, New Jersey, teachers worked with an expert in "resource-based instruction," David Loertscher, to revamp their approach to teaching research skills to elementary school students. Similarly, teachers at Ni?os Bonitos Elementary School in San Diego, California, used grant money to bring in experts in portfolio assessment, including Dennie Wolf and Grant Wiggins, to help teachers develop their new system of assessment. Though any direct impacts of these collaborations upon teacher appropriation of new performance assessments are unclear, teachers said that these experiences were positive.
An elaborate local-level collaboration among teachers and other educators to support assessment reform is taking place in New York City. There, the New York City Assessment Network — a consortium of three organizations, the Center for Collaborative Education, the Center for Educational Outcomes, and the Elementary Teachers Network — works with teachers at approximately 62 elementary schools across the city to implement the Primary Learning Record. These three organizations provide a range of professional development activities to participating teachers, including placing "teacher consultants" in the schools to assist in implementation, coordinating "study groups" led by these teacher consultants and other mentor teachers, and providing free continuing education classes at Lehman College for all interested participating teachers. Teachers suggest that the study groups, in particular, are a crucial element of the network that has helped them to use the PLeR effectively.
The successful introduction of new performance-based assessments can be facilitated or hindered by the interaction between the assessment reform and two other factors which inherently must either be compatible or at odds with the assessment reform. These factors are:
Exhibit 6-10 illustrates the relationship between the level of teacher appropriation of new assessment technologies and the coordination of assessment reform with other reform efforts, other assessments, both, or neither.
Coordinated Efforts to Reform Curricula or to Establish Standards
The coordination of assessment reform with the introduction of curriculum reform can be an important facilitator of assessment reform; similarly, the articulation of clear standards for content and performance can also serve as a facilitator of assessment reform. When teachers perceive coherence in the entire reform initiative, they are more likely to embrace each piece, including the new performance assessment; when the performance assessment is not clearly related to other reforms, or when these other reforms lag behind assessment reform, teachers remain cautious before they invest effort in coming up to speed on the new assessment. Coordinated Efforts to Reform Curricula or to Establish Standards
The coordination of assessment reform with the introduction of curriculum reform can be an important facilitator of assessment reform; similarly, the articulation of clear standards for content and performance can also serve as a facilitator of assessment reform. When teachers perceive coherence in the entire reform initiative, they are more likely to embrace each piece, including the new performance assessment; when the performance assessment is not clearly related to other reforms, or when these other reforms lag behind assessment reform, teachers remain cautious before they invest effort in coming up to speed on the new assessment.Exhibit 6-11 summarizes the parallel reform efforts in the areas of curriculum guidelines and content or performance standards experienced at each of the sixteen participant sites.
In some instances, curriculum and assessment reform have been coordinated efforts, and this coordination has furthered teacher appropriation of performance assessments. For example, teachers at Sommerville High School described how the Pacesetter mathematics program led them to overhaul their teaching practices for their Pacesetter classes. Though early on they modified their instructional practices to be compatible with the program, they realized only slowly that they had to modify classroom assessment practices to support the program's curriculum and instructional techniques.
Another example reveals simultaneously how coordination can facilitate, and lack of coordination can hamper, assessment reform. In Prince William County, Applications Assessments were developed to support implementation of the district's designated Standards of Quality, and coordination between these two reforms served as a facilitator of assessment reform. At the same time, however, the district implemented its Applications Assessments prior to completing its revisions to the district's curriculum — revisions were being made in all subject areas and at all grade levels. Though there is no inherent reason why assessment reform cannot precede curriculum reform and have a coherent system emerge, putting assessment reform first does leave teachers in a temporary quandary. Teachers at Westgate Middle School repeatedly asked, "How can we introduce this test when we haven't even decided on the curriculum yet?" Teachers also suggested that they would wait to see what the revised curriculum looked like before they invested a lot of time in adapting their classroom practices to support the new Applications Assessments. Note, however, that this example illustrates a short-term, and not necessarily a permanent, barrier to teacher appropriation of the new performance assessments.
Similarly, at Crandall High School, the absence of state-delineated content and performance standards left teachers involved in the development of assessment tasks feeling a bit rudderless. The fact that most of Oregon's Foundation Skills ran across subject areas exacerbated the problem, as teachers were unclear about how to connect them to the traditional content areas that continue to define how most high school courses are organized.
Compatibility with Other Assessment Requirements
Teachers also are more likely to appropriate performance assessment technologies that do not compete with other assessment requirements they face. Perhaps the clearest example of the challenge to appropriation that can occur if assessment systems compete comes from Manzanita High School. Teachers there are being asked to accommodate ASAP within the context of an already extensive district assessment program. Because ASAP represents no "added value" from these teachers' perspective (it is, they say, a performance assessment inferior to the one employed by the district), teachers view it as an add-on, not as a reform that can be coordinated with existing assessment requirements or that supplements the information yielded by those efforts in some valuable way.
Both Vermont and Kentucky explicitly designed their performance assessments to be coordinated with other more traditional and standardized components of an assessment system. In Kentucky, in particular, this coordination of assessments fostered teacher appropriation of the state's portfolios and performance tasks. Regardless of their particular criticisms of KIRIS, teachers at the participant Kentucky school say they believe the assessments — in combination as well as individually — in and of themselves have proven their value in the classroom.
4The paucity of data available on the financial costs of performance assessments (including costs of conducting professional development activities) makes it impossible for us to compare costs across sites.
5 Maryland, too, has a designated "test coordinator" in each school who functions in a capacity similar to that of Vermont's and Kentucky's network and cluster leaders. However, as has been described above, the impact of MSPAP professional development on teacher appropriation of the assessment at Walters Middle School is quite low.
-###-
[Chapter 6: Cross-Case Analysis 3: Part 1 of 3] [Chapter 6: Cross-Case Analysis 3: Part 3 of 3]