A r c h i v e d  I n f o r m a t i o n

Volume I: Executive Summary (cont'd)

While the Interim Evaluation of the Regional Educational Laboratories was successfully completed in accordance with the Standards, there are certain limitations to the study that must be addressed. These issues are discussed below, in part, to draw attention to important factors that the reader should keep in mind when reviewing the Lab-specific findings of the peer reviewers. This discussion will also provide the context within which the findings should be interpreted and serves as a caution against inappropriate use and overgeneralizations of the individual evaluation findings.

• The Interim Evaluation involved independent evaluations of each of the 10 Regional Educational Laboratories; it was not designed to be a cross-Lab evaluation of the REL program. Although a standard evaluation framework was used to guide the peer review process at each Lab, the Interim Evaluation was not designed to assess the Laboratory system as a whole. Instead, each Lab was evaluated independently as called for by the Standards.

• The evaluation of each Lab was based on a sample of products and materials; it was not based on an exhaustive review of the work conducted by the Lab in this contract period. Given the prolific nature of the Labs, it was neither feasible nor reasonable to conduct an exhaustive review of all of the products and services developed by each Lab in the past three years. Instead, DIR worked collaboratively with the Labs and ED to identify a sample of data sources which would give peer reviewers a sense of both the breadth and the depth of the Lab’s work in the current contract period. Consequently, peer reviewers were asked to evaluate each Lab on the basis of a very small portion of the work the Lab does. As a result, findings that suggest that a Lab could be doing more in a particular area may be an artifact of the sampling process that was used.

• Peer reviewers had a limited amount of time to assimilate large amounts of data and materials and learn about the work of the Lab. The Regional Educational Laboratories are complex organizations that provide a wide array of services and products in a variety of content areas. While some of the Labs are relatively new, others have been in existence for over 30 years. It is unreasonable to expect that peer reviewers (some of whom were unfamiliar with the REL program) could fully appreciate the nature of the Lab’s work and the context in which it was conducted, in the short period of time in which this evaluation was conducted. Given only a sample of products and services and a limited amount of time and resources to conduct this evaluation, peer reviewers were only able to "scratch the surface." This is not to say that peer reviewers did not have a good understanding of the materials they were asked to review but simply that those materials represent just a part of what the Lab does. For a more in-depth evaluation of each Lab, considerably more time and effort must be invested to allow peer reviewers the opportunity to explore more of the Lab’s work in greater detail and reflect upon what they have seen.

• Peer reviewers were selected to represent a broad spectrum of skills and expertise. While peer review panels were configured to include a variety of complementary perspectives, each panelist brought their own expertise to bear on the evaluation process. As a result, some panelists were able to address certain issues in more detail than others. For example, panelists who had a research background were more likely to focus on issues related to study design or analysis while those who had more administrative experience tended to focus more on the practical applications of the Lab’s work in a classroom setting. Although peer reviewers were asked to address each of the eight evaluation questions in their individual reports, their interest and ability to do so varied. Consequently, some reports include considerable discussion of certain issues of particular interest to the reviewer and only cursory coverage of other issues. Recognizing that individual panelists may focus on different areas of interest, the panels were purposefully constructed to ensure that diverse perspectives and experiences would be represented and that the panel as a whole would encompass the range of expertise needed for the evaluation.

• Although designed to be an interim evaluation, the review process was not very formative in nature. According to the Standards, an interim evaluation was to be conducted in the third year of the five-year funding cycle to assess each Lab’s performance and to provide recommendations for improvement. DIR’s contract to coordinate the Interim Evaluation was awarded in August 1998, just a few months before the end of the Lab’s third year of funding. Allowing time for the design of the evaluation framework and the recruitment and training of peer reviewers, the evaluation site visits were conducted in April and May 1999, almost half way through the Lab’s fourth year of funding. While the Labs reported that the panel’s findings could certainly help them in preparing their proposal for the next competition, many felt that the recommendations would be difficult to implement in the short time remaining in the current contract period. Further, some peer reviewers reported that they were hesitant to include recommendations in their evaluation reports without knowing the resource constraints under which the Lab was operating. To be truly formative, the evaluation needed to occur earlier in the funding cycle and include specific recommendations

Despite these limitations, the Interim Evaluation of the Regional Educational Laboratories yielded useful feedback to each Lab, including validation of areas in which the Lab was doing exceptionally well and identification of areas that need improvement. Some Labs also reported that the Interim Evaluation served as an opportunity to critically reflect upon their own work and assemble materials to chronicle the evolution of selected projects. One Lab even noted that this exercise was a good learning opportunity for staff, both old and new, to become more familiar with the work being done in their Lab. Others reported that the evaluation reports confirmed many of the issues already identified by their Lab as areas needing further attention and may now serve as leverage to initiate the change process. Overall, both peer reviewers and Lab staff reported that the Interim Evaluation was a positive learning experience.

Volume I: Executive Summary (Part 2)

Volume I: Executive Summary (Part 4)