A r c h i v e d  I n f o r m a t i o n

A Tool Is Only as Good as its Use

Using State Assessment to Enhance Capacity

The staff have used the state assessment in three main ways to foster capacity at the building level.

One use of the assessment was as a guide for curriculum development. Along with the frameworks, for example, CLAS was the main tool used by the English department to design its curriculum articulation to be used by all the families across the three grades (6,7,8). At the time of our data collection, a similar process was beginning in the math department as part of the Program Quality Review of the state School Improvement Program. CLAS was also providing useful insight into that process as well.

We place a strong emphasis on knowing all eight of the CLAS writing types and being exposed to different types of writing--poetry, scripts, articles, speeches. To know and understand the writing process--to understand writing as a process...

We use the frameworks and the CLAS test to map the styles of writing onto the grades at which grades--to introduce the styles, when to reinforce them, when they should reach mastery. For example, observation writing is difficult at grade 6 so we wait on that. Evaluation writing we start in sixth and then reinforce in seventh...

The math department in its preparation for giving CLAS discovered that they were not sufficiently preparing students in probability and statistics, one of the strands of the framework and an area assessed on CLAS. They were beginning to discuss how and when they might incorporate probability into the mathematics curriculum.

A second use of CLAS was to help develop pedagogical skills and improve instruction among the teachers in the school. "The CLAS test is a way of getting people to do more cooperative work and to teach in certain ways." For example, because CLAS incorporated open-ended mathematics problems, the eighth grade teachers (who would be giving CLAS to their students) were receiving assistance and modeling in the development and use of open-ended tasks for their students. They were thus able not only to help prepare their students for the assessment but to incorporate such tasks as a regular aspect of their instruction. They also shared these tasks and exemplar student responses with other math teachers in the department meeting observed during this study. Meanwhile, as a result of the articulation of the English language arts curriculum and the assessment, teachers in the English department were able to identify areas of the curriculum (like poetry) that weren't being addressed by some of the teachers and families. These areas were targeted for professional development, which led to a strengthening of instruction in the desired areas throughout the school.

For example, some [of the families] were not covering poetry.... What came out was that some teachers said I don't teach poetry (for example) because I don't understand it, don't know how to teach it.' So we did inservice on this.... Teachers who were strong in poetry teaching share with others how they did it. Teachers are also able to observe other teachers. [Our coach] is available to model lessons, and we have discussions about strategy.... Because of that, people are more comfortable with poetry; it's being taught now. We know this because when the students get to us [in the eighth grade] they know about personification and other aspects of poetry.

Finally, in this school, CLAS¹⁰ and preparation for CLAS had helped to generate a results orientation focused on student work. With the assistance of their coaches, the teachers had developed a pre- and post-test in writing and, more recently, in mathematics, given to all students at the beginning and then at end of every school year. Each year the writing prompt focused on one of the writing types assessed by CLAS and included in the curriculum articulation at each grade. Results from the pre- and post- test allowed the teachers to see growth, while the exercise also familiarized the students with the format and content of the state assessment they were to be given in grade 8. Results from this school-based process were then validated by the official CLAS results released during data collection for this study:

The CLAS scores that were released this year were generally very low. Two exceptions were mentioned in the newspaper article. One was [this school]. There were no scores of 6, but 23 percent were at a level 5 in writing. This was on an equal par with [the academic magnet high school].... Anyhow, the scores show that we are doing some things right here. We can take ordinary kids and help them achieve.

The scores in reading were not as high, so teachers were targeting this area as a major focus for improvement.

But this school-based assessment process, modeled on the state assessment, did more than provide evaluation information and test preparation. It also promoted the discussion of standards throughout the school and provided concrete professional development for performance-based assessment.

The schoolwide discussion of standards comes indirectly from CLAS. It has helped my teaching by focusing me on certain standards.... I really support the schoolwide discussion of standards and rubrics.... Within a school there can be tremendous variation. Teachers are influenced by the teachers they had. The expectations their teachers had of them will influence the expectations they have of their students.... In the schoolwide discussions we look at 500 essays and decide what's a 6 or a 5 and so forth. This has been going on at this school since I started. Everyone on the faculty is invited and gets paid to score. Some who might not do it otherwise come because of the pay, so the money is important. We do it after school and in the evenings.

In the previous section of this chapter we outlined a framework for describing teacher and school capacity. Using that framework, we can analyze the ways in which CLAS was used by this school as an avenue for capacity building. At the building level, it was clearly a vehicle for clarifying the vision, both in terms of the curriculum goals and in terms of the standards of performance expected of the students. It was also used through the department meetings, schoolwide pre- and post-testing, and targeted professional development activities as an avenue for increasing teacher knowledge and pedagogical skills. Finally, it contributed to building a school culture focused on improving student learning. To this end, CLAS and its mock assessments provided concrete measures of performance on which to judge progress and identify areas for improvement. Also, the coherence of the assessment with the learning goals of the school helped to promote teacher dispositions receptive to standards and to the change process.

It is important to note, however, that this school was able to make such extended use of the state assessment for capacity building only with the assistance of their coaches from the UC program. Without this additional guidance it would have been unlikely that the state assessment would have had such a positive impact on change at this site.

State Assessment and Capacity in Three States

Vermont. In Vermont, the portfolio assessment is seen and used as an expression of the statewide vision of reform. One important aspect of this is that the assessment is intended to put results at the forefront of reform effort, while leaving teachers and schools to decide how they will get there. In this respect, it models the theory of change underlying much of the reform in that state (see Chapter 3). For the teachers whose students compile the portfolios (grades 4 and 8), the assessment has also been an opportunity to learn about the expected outcomes in writing and in math. This has been particularly important in math because the goals in this area are new for most teachers.

The reform I'm most familiar with is the portfolios, which I think has significantly influenced my math and writing instruction.

[Portfolios] are the state's way to get teachers to change, and teachers have griped. But the portfolios are in place much better. The program was [initially] disjointed and teachers felt it was imposed on them.... But it's an effective way to get teachers to change and I embrace the NCTM standards.... I might not have done it otherwise.

To foster teacher learning in connection with the portfolio assessment, the state has sponsored professional development through workshops and networking.

Math in the portfolio assessment was so totally new to me that I really needed inservice work. In the first year of the portfolio program, I took a week long summer institute. At the end of the week, the other teachers and I were still wondering what we were doing. Now I see this as a birth of a new way for me to think about math. I am now a network leader, where I get to talk to other teachers about math instruction.

The state portfolio workshops have been very helpful for many teachers.

The inservices have influenced my attitude toward time spent on basic facts vs. problem solving.

I don't drill as much on basic skills anymore and we changed our textbook to one that lends itself to portfolios and the NCTM standards. We increased the amount of material to teach [and] it used to be all skill and computation, though I did a lot of estimation [too].

The portfolios and the professional development activities associated with them seem to have served as a means for increasing teacher knowledge and, at least for some, of engendering their support for the direction of the reforms. In addition, the move to use portfolio assessment in state certification and program approval means that new teachers will have gained some knowledge and experience developing portfolios themselves during their preservice training. This should add to their ability to use them effectively with their own students.

The portfolios have even been a way of developing knowledge and support among the general public and potential partners in the reform, as problems and debates about the new assessments have been shared openly. Parents are also supposed to see their students' portfolios, which are intended to make more vivid for non-educators as well as for educators what the reforms are trying to do. Such a very public approach to building understanding and capacity among the broad spectrum of stakeholders is seen as crucial for ensuring its long-term stability and success.

Despite these contributions to capacity made through and in connection with the portfolios, however, some policies and practices surrounding the assessment have mitigated its effectiveness as an avenue for building either individual or organizational capacity. Perhaps the greatest shortcoming from the perspective of the individual teachers involved is that although they have learned about performance assessment and about the reform goals, some teachers feel they have received little assistance either through the networks or inservices in making the link to instruction. One reason for this may be that the assessment is still very new and reliability on scoring the portfolios has been elusive; because of this, the focus of the network meetings seems to have been largely on scoring with less attention to instructional applications:

The inservices on portfolios were helpful in learning about assessment, but not for instruction.

The implementation differs from teacher to teacher. They can teach writing how they have always done, they just have to do more of it.

In addition, respondents noted that there had been little follow-up on the inservices they attended.

Teachers also complained that the time required for scoring the portfolios, in both math and writing, was taking them away from working on instruction. This seems to imply that the respondents did not see the scoring process or the information they received from it about their students as instructionally relevant or at least as sufficiently relevant--to justify the time commitment. A contributing factor was that both math and writing were given in the same grade and those teachers felt overburdened.

This last fact seems also to have hindered the use of the portfolios to build organizational capacity in the school. At the time of our data collection, the staff development and networks were geared largely to the fourth and eighth grade teachers without much broad schoolwide involvement. Again, this may be due to the early stage in the portfolio development and use, but it is also characteristic of traditional models of professional development, which are aimed at individual teachers rather than schools and other organizational units. It will likely require concerted effort to move beyond this model to one which uses the assessment to foster capacity at the broader organizational level. Some of the respondents in this study had already recognized this as a problem and were moving to spread out the portfolio work beyond the initial two grades.¹¹

A final note on the limitations of the portfolios thus far is that while teachers used them as indications of the kind of writing or mathematics to stress, our respondents did not talk about actually using either the scores (which were not yet reliable) or any information about their students' performance gleaned from their own scoring of the portfolios. To the extent that assessment results are not used by teachers, the intended results orientation of the reform appears weakened.

Michigan. As in Vermont, the state assessment in Michigan is also an expression of the state vision for reform. Indeed, with no curriculum frameworks (like California) nor an articulated vision statement (like the Common Core in Vermont), MEAP is the main vehicle for communicating the goals in reading and mathematics, the two curricular areas investigated in this state. The Essential Goals and Objectives on which MEAP is based have long reflected a meaning-centered approach to reading and have also been revised to more closely reflect the NCTM standards in math. The potential usefulness of the assessment as a means of building capacity is enhanced by the fact that the Essential Goals and Objectives are very clear and are open to the public.

Moreover, unlike Vermont, Michigan has instituted a number of policies that have the potential of strengthening the impact of the assessment on organizational capacity. One area where this is the case is curriculum development. The Essential Goals and Objectives are the basis for the state's Model Core Curriculum Outcomes, which in turn are to serve as the basis for district core curriculum. The second area is school improvement. According to state law (PA 25) each school must develop school improvement plans and write improvement goals focused on student outcomes. Because MEAP scores cover several core curriculum areas and must be publicly reported, schools have tended to use these to set some of their improvement goals. Thus the state assessment is providing useful information to schools that assist them in targeting areas for improvement, but the reforms leave discretion at the school site to identify exactly what those goals will be and how they will be achieved. In this respect one could view the assessment as contributing to school capacity by being a resource at the disposal of local school personnel. It has also been the focus of some staff development, primarily in the form of workshops to familiarize teachers with the content of the revised goals and objectives.

Yet MEAP is actually rather limited in its usefulness and use as a tool for capacity building. Unlike Vermont, this limitation derives largely from the nature of the assessment itself, which remains almost entirely multiple choice. This traditional format has several implications. While the content assessed by the MEAP is consistent with the NCTM standards, the assessment is inadequate to fully reflect those standards or the approach to mathematics that underlie them. The reliance on multiple choice, for example, continues to stress a "right answer" approach to mathematical problems; it allows students neither to demonstrate their ability to communicate about mathematics (one of the NCTM standards) nor to show their reasoning on complex problems. One respondent in a Michigan elementary school discussed the negative impact of MEAP in her school.

We were trying to move lower elementary teachers into a more progressive plan, when MEAP came in and said,you can't do developmentally appropriate things.' MEAP sort of sent things backwards. The timing and the way it came down was unfortunate because we were on our way to being more constructivist and developmental. But we are trying to respond to the mandates.

This limitation also means that it is not a very strong tool for teacher learning about new approaches to mathematics. Unlike CLAS, which included extended response items, or the Vermont portfolios, which include even more extended examples of student work, MEAP does not provide teachers with any picture of what actual student work that reflects the standards looks like. Thus, while the results on MEAP may help schools and districts to target areas of deficiency, it does not serve to focus attention on the quality of student work. Moreover, there is little reason for teachers to be involved in scoring,¹² which is handled by the state. The disjuncture between assessment and instruction, which is so characteristic of American education, is thus maintained, and teachers' knowledge and skills are not substantially increased.

California. If the Vermont and Michigan assessments provide examples of missed opportunities for capacity building, California stands in a league all its own in this regard. The strength of CLAS with regard to capacity is that unlike the MEAP, the content and format of CLAS guided teachers toward new ways of looking at content and a basis for thinking about instruction. Those respondents who had had the opportunity to become familiar with CLAS were very supportive, stating that it was "a wonderful assessment" and "going in the right direction." There were a few teachers who felt that it did not go far enough--for example that the lack of opportunity for revision in the writing assessment meant that it did not truly reflect a process approach to writing. But even these teachers were mainly supportive. In addition, respondents noted that the involvement of teachers in scoring the assessments provided them an excellent learning opportunity.

But while some teachers--like those in the middle school described earlier or those who participating in the scoring--had the opportunity to become familiar with CLAS and use it as a learning tool, the vast majority of teachers and other school personnel were not so fortunate. Instead, an emphasis on secrecy to protect reliability, coupled with management errors, meant that most teachers and districts remained unfamiliar with the actual content or format of the assessment--even up until the time it was administered. And if the educators were in the dark, the public was even more so. Opponents of the reform were able to use this situation to engender vocal opposition, which was in turn used by the governor to kill the assessment, thus garnering conservative political support in his bid for reelection. It should be noted that the very short time table for development of the assessment imposed on the California Department of Education (CDE) by the governor was a major contributing factor. Yet, CLAS, even in its developmental stages, provided a potentially powerful tool for teaching the public and California educators about the concrete goals of the reforms and the type of learning and performance students are being asked to do. Failure of the CDE to focus on this use of CLAS left both the assessment and the reforms vulnerable. It is not yet clear what the long term ramifications of this mistake will be.

One lesson from the examples of state assessment in California and Vermont is that the use of state assessment as an instrument of accountability may be in tension with its potential use as an instrument for teacher and system learning. Accountability requires a high degree of reliability. In Vermont, with its limited time and resources, this meant limited attention on using the assessment to improve instruction. In California it engendered a level of secrecy that ran counter to building either capacity or support among a broader spectrum of the public or school personnel. That such would be the results of that tension is not, we believe, a foregone conclusion. A consistent and strategic emphasis on capacity building may have--and may still--lead to alternative scenarios.

A second, and broader, lesson from all three states is that both the design and the strategic use of the systemic tools discussed above can increase or decrease their effectiveness for capacity building. This seems to be true in areas other than assessment. Below we discuss some findings with regard to an area traditionally associated with capacity building, that of professional development.

¹¹ It might be recalled that in the example of the California middle school above it was also the as school, rather than the state or the district, that designed a means for using the assessment to push forward standards development and learning beyond the grades assessed.

¹² Note: MEAP now includes three open-ended questions in math which can be scored at the discretion of the district. Neither of the sample districts in this study appeared to have availed themselves of this option.
-###-

[Systemic Tools to Enhance Capacity]

[A Tool is Only as Good as its Use (continued)]