A r c h i v e d  I n f o r m a t i o n

Sample and Data Collection

The sample for this study consisted of seventeen schools from eight locations. Seven of these are districts in the U.S., including: Bellevue, Washington; Chicago, Illinois; Denver, Colorado; Jefferson County, Kentucky; Milwaukee, Wisconsin; Rochester, New York; and Sweetwater, California. One high school and one elementary school were included from each of these districts except for Sweetwater, which is a high school district. The eighth location was Victoria, Australia, from which two high schools and two elementary schools were included. These venues were selected because of their reputation for having strong school-based management plans, including significant decision making authority at the school level. Phone calls were made to district officials to verify the strength of their decentralization plans. The specific schools studied in each site were selected based on information provided by district officials and/or researchers familiar with the site that significant curriculum and/or instructional reforms were underway at these schools. The intent was to include exemplary schools in the sample so as to enhance the likelihood that such reforms would in fact be found.

Prior to beginning data collection, all members of the research team attended a two-day training session. Two members of this team visited each school for two days, during which data were collected through structured interviews. Interviews focusing on school-based management and school innovations in curriculum and instruction were held with administrators, teachers, community members, and (at high schools) students. Included in the set of interviewees were members of the governance council and other participative structures, department heads, the union representative, teachers who have been actively involved in the design, adoption, and/or use of innovative practices, and teachers who have not been involved in the innovations at the school. The number of interviews conducted at the schools ranged from 13 to 24, with an average of 18. Interviews typically lasted forty-five minutes to an hour.

Variables, Data Coding, and Analysis

The study examined seven organizational variables and four areas of innovation to see how they were related. The seven supporting organizational conditions are power, knowledge, information, rewards, instructional guidance, leadership, and resources. The innovation areas are teaching for understanding, use of technology, educating all students, and integrated approaches. For each of these variables, a relevant set of questions was determined, along with potential categorical responses to these questions. These are identified in Appendix A. The questions are based on findings from the first, exploratory phase of the study. They address aspects of each domain that seemed, based on the qualitative analyses in phase one, to make a difference in whether the school was employing SBM to introduce changes in instruction and learning. For example, the power variable included questions about areas of influence, involvement of various stakeholders, and numbers and kinds of forums in which decisions are made. The resources variable included access to external grants and extension of resources by creating partnerships with community and business groups.

To code the variables, a qualitative data base consisting of the responses of all the interviewees at each school to each question was constructed. A coding scheme was then developed with which to code the seven supporting conditions and four types of innovations. For each school, two coders read the full set of interview responses and then assigned a rating for each question. One member of the research team coded all seventeen schools, while "second coder" duties were divided among five additional members of the research team. When possible, each pair of coders included at least one person who had gone on the site visit to that school.

Prior to coding, all coders participated in a workshop in which the research team members provided descriptions of the schools they had visited, including an overview of the SBM governance mechanism and the nature of the reforms taking place. This workshop reinforced the earlier training session and enabled coders to develop a common understanding of the variables being assessed in this study as well as the range of differences on these variables exhibited among the seventeen schools in the sample. A shared understanding of the variables provided guidance to the coders regarding the type of information that was relevant to answer the coding questions. Familiarity with the range of characteristics within the sample was necessary to enable coders to use a similar frame of reference for assessing each individual school. This is because they were asked to answer the coding questions relative to the schools in this sample only rather than relative to the full spectrum of schools in general.²¹

After the coding process was completed, points were allocated to the responses for each question (e.g., zero points for "low," one point for "medium," and two points for "high"). For each school, a score for each variable (for each coder) was calculated as the sum of the points for the responses to the relevant questions.²² To assess the level of "interrater reliability," Spearman rank correlation coefficients between the two sets of scores for each variable were calculated.²³ These correlations are as follows: power -- .80; knowledge -- .85; information -- .65; rewards -- .32; instructional guidance -- .56; leadership -- .73; resources -- .65; teaching for understanding -- .89; use of technology -- .78; educating all students -- .64; and integrated approaches -- .48. While most of these are adequate, the correlations for instructional guidance and integrated approaches are marginal and the correlation for rewards is poor.²⁴ While we decided not to drop any of these three from the analysis, results for these variables should be interpreted with caution.

We did not necessarily expect these measures to have high internal consistency, because the dimensions comprising them can vary independently. For example, on the resources variable, schools can obtain outside grants but not community partnerships. Instead, we conceptualized these variables systemically; i.e., in systems there are different routes to the same outcome (e.g., Beer, 1980). Thus, our primary interest was in whether the total presence of multiple aspects of each variable makes a difference in the school's innovation adoption activity. This is also consistent with earlier exploration of the impact of high involvement, where scales examining the impact of power, information, knowledge and skills, and rewards were comprised of the sum of a number of practices and the extent of employee involvement in each (e.g., Lawler, Mohrman, & Ledford, 1992). This approach does not deny that some dimensions may be more important than others. Given the size of our sample, we cannot explore those dimensions with great confidence, but we present observations on the patterns that we can detect.

To analyze the data, the scores from the two coders for each school were averaged to generate a single index for each of the variables. Since the primary hypothesis of this study was that more curriculum and instructional reforms will take place when more of the supporting conditions are present, an analytical technique was needed that would examine the patterns of findings across all variables simultaneously. Since the small sample size limited the feasibility of using more sophisticated statistical analyses (e.g., regression), an informal pattern analysis was utilized to evaluate these patterns. For this analysis, variable indices were dichotomized into "high" and "low" scores. This was done simply by determining whether the score for a variable at a particular school was above or below the mean of the distribution of the scores for that variable.²⁵ Patterns reflecting high and low levels of these supporting conditions and reforms were examined to assess support for the basic hypothesis underlying this research.²⁶

²² For example, if a coder rated all four questions associated with the Teaching for Understanding variable as "considerable", which is worth two points, the score for that variable would be eight.

²³ To calculate these correlations, the scores from the each coder were rank ordered and these ranks were then correlated with each other. Rank order correlations were used rather than normal correlations since our primary analysis, as indicated below, is not based on the specific variable scores themselves but instead is based on a distinction between relatively high and low scores. In fact, the process of calculating variable scores was not originally intended to provide a precise measure of these variables, but simply was intended as a means by which to identify those schools that were high and those that were low on each variable. Therefore, it was more important that coders agree on the relative ranking of the schools than on the actual scores.

²⁴ The low correlation for rewards is largely due to the discrepancy in the ratings of three schools by two coders. In the absence of much information in the interview response data, one coder rated each of these schools very low on this variable. In contrast, the second coder, who had visited the schools as part of the research team and thus had greater familiarity with them, coded the variable considerably higher. (This was the only variable, and the only schools, for which obvious and consistent discrepancies existed between the two coders scores.) In addition, the amount of variation in the scores for the rewards variable, and also the instructional guidance variable, is somewhat limited, which may have contributed to the lower correlations for these variables.

²⁵ For example, the scores for the Knowledge variable ranged from a minimum of 1.5 to a maximum of 5.5, with an average of 3.5. If the score for School A were above the mean, it would be coded as "high;" if it were at the mean or below, it would be coded as "low."

²⁶ An important question has to do with the validity of the measures we use in the analysis. In other words, to what extent can we be confident that a school really has in place the level of the supporting conditions or reforms indicated by our measures? One particular concern could be that the total amount of reform taking place at these schools is underestimated since we limit our focus to only four categories of innovations. However, there is reason to believe that this is not a problem. In the first phase of our research, responses to open-ended questions regarding the types of reforms being implemented at the schools fit primarily into these four categories, indicating that these were the most popular innovations taking place. Although we targeted interview questions about these reforms in the second phase of the research, we also asked open-ended questions about other types of reforms being implemented. As these yielded very little additional information, it is valid to conclude that there was not a significant amount of other kinds of reform taking place.

More generally, we have reasonable confidence in the validity of our measures for a number of reasons. First, they are based on information that came from a wide variety of sources at each school, some of whom were uninvolved in the reforms being addressed and thus had no incentive to exaggerate the extent of the reforms. Second, the fact that we found variation across the schools in our sample on most of the variables suggests that there was no widespread social desirability bias at work that led all respondents to be overly optimistic about the level of the supporting conditions or reforms. Finally, the nature of our measures -- dichotomous ratings of the relative level of any particular variable -- are sufficiently broad that we are quite confident that the schools rated high on a particular variable do in fact have a higher level of that variable than the schools rated low.
-###-

[Part 3: Focus of the Study]

[Part 5: Results]