A r c h i v e d  I n f o r m a t i o n

Volume I: Executive Summary (cont'd)

1. Evaluation Design

This section provides a brief overview of the evaluation design implemented in the Interim Evaluation of the Regional Educational Laboratories. The section begins with a review of the evaluation framework, including the evaluation criteria and evaluation questions. The major components of the peer review process, including recruitment, panel assignment, and training, are then discussed. The section concludes with an overview of the evaluation process as it was implemented, including a discussion of individual evaluation plans, on-site evaluation visits, and the reporting process. For a more detailed discussion of these issues, see Volume II.

1. Evaluation Framework

Although the Standards call for each Laboratory to be independently evaluated, DIR developed a standard evaluation framework to be used by the peer review panelists at each Lab. The framework includes a set of evaluation criteria against which the Lab’s products, services and activities were assessed through a series of overarching questions. For each of the evaluation questions, a set of evaluation indicators were identified to guide the peer review panelists. The framework also includes a list of possible data sources that could be used to evaluate the various products and services of the Lab. Using this standard framework, DIR developed an individual evaluation plan for each Laboratory, selecting specific activities and data sources to be reviewed that were representative of the work of that particular Lab.

The evaluation framework is guided broadly by the four evaluation criteria specified in the Standards:

• Implementation and Management
• Quality
• Utility

• Outcomes and Impact

The Standards further provide guidance on how peer reviewers can assess whether a recipient meets the evaluation criteria, with specific guidance provided for assessing the performance of the Regional Educational Laboratories.

Eight evaluation questions were developed to correspond to the criteria set forth in the Standards and to broadly reflect the work of the Labs. The evaluation questions were designed to allow peer reviewers to evaluate the Lab’s products and services by addressing the following questions:

• To what extent is the REL doing what it was approved to do during the first three contract years?
• To what extent is the REL using a self-monitoring process to plan and adapt activities in response to feedback and customer needs?
• To what extent is the REL developing high quality products and services?
• To what extent are the products and services provided by the Laboratory useful to, and used by customers?
• To what extent is the REL focused on customer needs?
• To what extent is the REL’s work contributing to improved student success, particularly in intensive implementation sites?
• To what extent does the REL assist states and localities to implement comprehensive school improvement strategies?

• To what extent has the REL made progress in establishing a regional and national reputation in its specialty area?

For each of the eight evaluation questions, a set of evaluation indicators was developed to provide guidance to the peer reviewers regarding specific factors to consider in the course of the review process.

While the Interim Evaluation was designed to assess Laboratory performance in the first three years of the current contract period, it was neither reasonable nor feasible for panelists to review all of the work conducted to date. Instead, peer reviewers were asked to address the evaluation questions based on a review of selected data sources sampled from each of the following categories: (1) "signature works," (2) additional selected outputs, and (3) Laboratory operations materials. Possible data sources included printed and electronic materials, in addition to on-site interviews and observations.

• Signature Works. For the purpose of this evaluation, a "signature work" was defined as a significant illustrative work (e.g., major product, strategic thrust, or service package) that best exemplifies the work of the Laboratory in the current contract period. While it was understood that the signature works may not represent a typical cross section of the work conducted at a Laboratory, their inclusion was meant to provide peer reviewers with a more in-depth look at some of the major program areas of the Laboratory.

• Additional selected outputs. In addition to the selected signature works, peer reviewers were provided with a sample of other programs, products, and services of the Lab. This review of other outputs was intended to provide peer reviewers with a broader base upon which to gauge the totality of activities of the Lab under the OERI/REL contract.

• Laboratory operations materials. The Standards mandate a series of data sources that address how the Lab operates at the organizational level. In addition to the required items (e.g., original contract proposal, contract modifications, quarterly reports, annual plans and updates), Labs were asked to identify other materials or data that would allow the peer reviewers to address questions of implementation and management. These other data sources often included organizational charts, Board rosters and minutes, and REL funding and staff allocations by task.

2. Peer Review Process

According to the Standards, each peer reviewer must have the necessary knowledge and expertise in the area of the project being reviewed to evaluate the performance of a recipient. This experience may include:

• Expert knowledge of subject matter in the area of the activities to be reviewed
• Expert knowledge of theory or methods or both in the area of the activities to be reviewed
• Practical experience in the area of the activities or type of institution or both to be reviewed
• Knowledge of a broad range of educational policies and practices
• Experience in managing complex organizations
• Expertise and experience in evaluation theory and practice

In addition to these characteristics, peer reviewers must also be free of any conflict of interest.

In an effort to attract a broad pool of qualified peer review candidates, DIR used a multi-pronged approach to recruitment. Nominations were solicited from a number of sources, including the Lab Directors, the U.S. Department of Education, DIR’s Technical Work Group, and a variety of educational, business and professional organizations. Additionally, a "snowball" approach was implemented by asking candidates to recommend others who may be interested in serving as a peer reviewer.

Close to 300 peer reviewer nominations were received. A resume or vitae was obtained from each potential candidate and reviewed for relevant experience and expertise. A standard interview protocol was also completed with each candidate via telephone to gather additional information on the individual’s areas of expertise, screen for any real or perceived conflicts of interest, and discuss the responsibilities of serving as a peer reviewer in this Interim Evaluation.

After seeking feedback from ED and the Labs, a final pool of peer reviewer candidates was submitted to the Assistant Secretary of OERI for approval. It was from this group of highly qualified individuals that the actual reviewers were selected and assigned to Lab-specific review panels.

Because the Standards call for independent evaluations to be prepared by each peer reviewer, DIR felt that it was important that a broad range of perspectives and expertise be represented on each Lab review panel. While panels were tailored to match the unique characteristics of each Laboratory, each panel was configured to contain, at a minimum, one or more individuals with the following characteristics:

• Research expertise

• Evaluation methodology expertise
• Practical experience (e.g., educational practitioner)
• Specialty area expertise (in the specialty area of the Laboratory)
• Knowledge of the Labs or Lab-like activities
• Experience managing complex organizations
• Educational policy & practice expertise

• Knowledge in content areas of Lab focus in addition to that of the specialty area

Although 40 peer reviewers were initially assigned to fill 60 panel slots (i.e., six panelists at each of 10 Labs), last minute attrition resulted in the loss of several reviewers. In the end, 34 reviewers actually conducted evaluation site visits, with half of the Labs being evaluated by a five-member panel and half being evaluated by panels of six members.

The training provided to review panelists was designed to ensure a common base of understanding of the evaluation framework, the peer review process, and reporting responsibilities. The peer reviewer training was conducted through a series of three teleconferences, with a total of five hours of training provided. Each training session was focused on a different aspect of peer reviewer preparation. Training Sessions #1 and #3 were designed to be Lab-specific, while Training Session #2 was a general session on the evaluation framework.

In addition to the basic training provided to all peer reviewers, panel chairs were provided with an additional one-hour session to review the roles and responsibilities of panel chairs, including facilitation of Training Session #3, and to clarify any procedural issues. DIR staff also met with the panel chair on-site before the evaluation visit began to answer any last-minute questions and to review expectations.

3. Implementation of the Evaluation

The review process began with the development of an individual evaluation plan for each of the 10 Labs using the standard evaluation framework previously described. Once a plan was approved by ED, DIR obtained advance materials from the Lab for distribution to the peer reviewers prior to the on-site visit. Evaluation site visits were conducted in April and May of 1999. During each visit, panel members had an opportunity to interview Lab staff, partners, and clients and review additional data sources not included in the advance materials. At the end of the visit, each peer reviewer was expected to submit an individual evaluation report. Following each visit, the panel chair prepared a synthesis report highlighting the findings of the panel members. All reports were shared with the Lab, who was then given the opportunity to submit a written response.

Using the standard framework developed for this evaluation, DIR worked collaboratively with ED and the Labs to develop an individual plan that included data sources that were representative of the work conducted by the Lab under the current REL contract. Each evaluation plan included three clusters of data sources: (1) signature works, (2) selected inventory items, and (3) Lab operations.

• Signature Works. Of the five or six signature works nominated by the Lab, two were selected for inclusion in the individual evaluation plan. When selecting signature works, DIR was guided by several factors. Collectively, the works selected had to:

• Represent the major functional areas of a Lab (i.e., R&D, field services, and strategic alliances)
• Include at least one work being implemented in the field
• Represent the specialty area, if possible
• Have adequate, yet manageable data sources available for review

• Other Selected Outputs. After selecting the signature works, DIR, in collaboration with ED and the Labs, purposefully selected items from the inventory. Collectively, these items were meant to:

• Provide added depth to the overall evaluation of Laboratory products, programs, services and complement the other work that was reviewed

• Provide breadth to the evaluation by filling in gaps not covered by the selected signature works and Laboratory operations

• Laboratory Operations. In addition to the data sources mandated by the Standards (e.g., original contract proposal, contract modifications, quarterly reports, annual plans, and updates), Labs were asked to supply data and materials that reflect how work is accomplished at the organizational level (e.g., organizational structure, funding allocations, needs assessment processes, quality assurance mechanisms, and self-evaluations).

Advance materials were shipped to the peer reviewers approximately two weeks prior to the scheduled site visit. The shipment included copies of the data sources specified in the individual Lab evaluation plan, as well as a briefing booklet prepared by DIR. The briefing booklet included a copy of the individual Lab evaluation plan, an agenda for the site visit, a list of site visit participants, review worksheets, the data matrix, a format for the evaluation report, and the inventory of products and services prepared by the Lab. Peer reviewers were expected to review the set of advance materials prior to the site visit and submit any questions or additional data requests to the panel chair one week before the visit.

Evaluation Site Visits

The Interim Evaluation of the Regional Educational Laboratories consisted of a review of hard copy and electronic materials prior to arrival on-site and additional data collection, including in-depth interviews, group interviews, and staff presentations conducted on-site. Local site visits to observe activities and interview partners and customers were also conducted in some cases. The time spent on-site was intended to give panel members an opportunity to meet with key Laboratory staff and speak with individuals (in person or by teleconference) who are consumers of the Lab’s services (e.g., teachers, administrators, parents), as well as Laboratory partners and collaborators. The on-site review was expected to provide reviewers with a much richer understanding of the Laboratory than could be gleaned from reviewing products, programs, or reports alone.

Peer review panelists spent five days at the Laboratory site. Approximately half of that time was spent conducting Laboratory data collection. During the remaining time, peer reviewers met as a group to discuss their impressions and findings and to draft their independent evaluation reports.

According to the Standards, each peer reviewer is expected to prepare an independent report on his or her assessment of the quality of the work reviewed according to the evaluation criteria. In addition to identifying the strengths at each Lab, the Standards specify that the reports should include any recommendations the peer reviewer may have for improving the recipient’s performance. To facilitate the review process and ensure consistency across peer reviewers and Labs, DIR developed review worksheets and a standard reporting format to be used by panel members in their individual evaluation of the Laboratory.

In addition to the individual evaluation reports prepared by each of the peer reviewers, the Interim Evaluation of the Regional Educational Laboratories also called for a synthesis report prepared by the panel chair. The objective of the synthesis report was not to provide a consensus, but rather to identify common themes and highlight the range of findings across reviewers. The synthesis report also provided a listing of the major recommendations for improvement suggested by the panel members.

Individual evaluation reports were due at the end of the site visit. A draft synthesis report was due two weeks after completion of the site visit. The draft synthesis report was distributed to all panel members for comment. The panel chair then submitted a final synthesis report to DIR, reflecting the feedback provided by panel members on the draft report. After DIR reviewed the reports for completeness and clarity, each Lab Director was sent a package containing the synthesis and individual reports for his or her respective Lab. The Lab was given approximately two weeks to review the reports and submit a written response for inclusion in the DIR final report.

Volume I: Executive Summary (Part 1)

Volume I: Executive Summary (Part 3)