Developing Effective Performance Indicators for Different Learning Environments

Purpose

During the past few years, higher education found it increasingly necessary to explain and defend itself to a variety of internal and external constituencies. Such demands for assessment and accountability resulted in a growing trend toward the use of performance indicators. However, there existed no model or process for the regular and public assessment of the degree to which a university and its agencies were achieving the state's goals and fulfilling priorities. The primary purpose of this project, using three campuses of the Texas A&M University System (TAMUS), was to develop and disseminate such a model and process, along with a list of valid, explicit, quantifiable performance indicators suitable for longitudinal analysis of achievement and productivity.

Innovative Features

It was agreed at the onset of planning that any accountability system developed must champion the cause of multiple missions and advocate the principle that differences in mission do not imply differences in quality. By recognizing and advocating a variety of emphases in mission the TAMUS model responded to individual campus missions and rejected the traditional conflict in priorities among teaching, research, and service. Accompanied by a core of indicator activities that all institutions must perform, the model contributes to the accountability that flows from clarity, fulfillment, and measurement of individual institutional mission.

In early 1996, TAMUS set about to assess performance within its ten universities and eight agencies by developing an accountability system that included reporting periodically on the results achieved in relation to goals assigned and resources received from its public benefactors. Guided by the principle that consensus and shared ownership are crucial in developing performance measures, the chancellor initiated a participatory process for developing the core of indicators that would apply systemwide, researching measures to ensure their validity, providing options for individual institutions and agencies, and involving faculty leaders.

Four systemwide task forces were established. Membership from the universities and agencies included faculty senate leaders, senior- and mid-level administrative staff, from chief executive officers to student affairs, and lay members from advisory groups within TAMUS. The charges for the four task forces were as follows: develop a TAMUS response to changing state needs, develop or refine processes, as needed, to ensure effective participation by external interests in targeting teaching, research, and service, develop a list of valid, explicit, quantifiable performance measures suitable for longitudinal analysis of achievement and productivity, and develop guidelines for financial incentives that are linked to performance measures.

The task force recommended the following key accountability areas and indicators for evaluating and improving academic quality in TAMUS. The task force also recommended that certain core indicators in these areas (indicated by asterisks in the list below) apply to all members.

1. Mission Focus and Fulfillment
• Development of an approved strategic plan to support the mission statement *
• Demonstrable attainment of goals contained in the approved plan*
• Institution's or agency's contribution to the state of Texas or region*
• Percentage of total dollar value to historically underutilized businesses*
• Institution's or agency's contribution to the state of Texas Higher Education Planning Benchmarks, 1997-2001
2. Quality of Faculty
• Academic and other credentials of faculty
• Ratio of full-time faculty to full-time employees
• Accreditation of degree-granting programs
• Average annual salaries for full-time faculty by rank
3. Teaching Quality and Effectiveness
• Percentage of lower-division classes taught by tenured or tenure-track faculty*
• Percentage of students completing courses*
• Graduate or current student satisfaction with the instruction provided (for example, student opinion surveys and "consumer feedback")
• Graduation rates of first-time, full-time, first-year students*
• First-year retention rates, broken down by student demographic groups*
4. Student Success
• State licensure examination pass rate (for example, EXCET and PE exam)
• Employers' satisfaction with graduates
• Number or percentage of graduates who have employment in their choice fields of graduation
• Degrees granted by levels
• Pass rate on Texas Academic Skills Program categorized by student demographic groups
• Percentage of graduates accepted to graduate and professional schools by the time of graduation
5. Financial Responsibility and Efficiency
• Net total current fund revenues to total current fund revenues*
• Net auxiliary enterprise revenues to total auxiliary enterprise revenues*
• Administrative cost as a percentage of total institutional expenditures*
• Space utilization ratios of classrooms and laboratories*
• Education and general expenditures per full-time equivalent (FTE) student
• Education and general expenditures per credit hour
• Plant equity to plant debt (ratio)
• State appropriations as a percentage of total revenues
• Total private gifts as percentage of state appropriations
• Annual education and general expenditures per degree awarded
6. Research Funding and Quality
• Annual amount of research dollar expenditures*
• External or sponsored research funds as a percentage of state appropriations
• Number of dollars in non-state, nonlocal research support
• Ratio of non-state-to-state research dollar expenditures
• Annual income from patents, licenses, agreements, and copyrights
• Annual research dollar expenditures generated per FTE graduate student
7. Institution and Agency Cooperation and Collaboration
• Number of articulation agreements with community colleges
• Number of transfer students to and from the institution
• Number of collaborative projects with other state institutions and agencies
• Number of institutional partnerships with K-12 school districts
8. Public Service
• Institution's or agency's level of service to Texas residents as measured by enrollment in extension and service programs (for example, distance education, arts, and health-related services)
• Public service hours of library (number of hours per week library open)
• Number and dollar amount of institution's or agency's resources, services, and expertise made available to meet a community need outside the context of the instructional and student services program

* Indicators currently required of all state institutions by the Texas legislature

Each TAMUS institution and agency first prioritizes the above eight accountability areas and then the mission-specific indicators. The task force recommended that the institutions and agencies limit their indicators to a total of 20. The primary philosophy is continuous performance improvement and value added through time, rather than performance comparison. The TAMUS institutions and agencies are currently working within these parameters to come up with their respective models.

Evaluation

In addition to the involvement of several system-wide committees, the ultimate evaluation of the model and the process will be its adoption, all or in part, by TAMUS and the Texas state government. Beginning in December of 1998, the Texas State Legislature required all institutions to formally report and comment on their performance for 1996, 1997, and 1998-the first of an ongoing set of hearings concerning results of the indicator model. At present, the effort is to secure more buy-in for the model among internal and external stakeholders, particularly campuses in the TAMUS.

Numerous publications and presentations which flowed from the project and the reaction from such events gave the project validity, as did professional evaluation and acceptance of the results by a myriad of international, national, and regional forums and publications. However, the primary indicator of the project's success, set out in the original proposal, was the general acceptance of the model by the state of Texas.

Project Impact

The movement toward developing a formal system of accountability constituted a pre-emptive strike designed to heighten the effectiveness of TAMUS and improve the public's and policy makers' understanding of and confidence in TAMUS and higher education in general. By developing such a pre-emptive strike, TAMUS managed to forestall possibly harsher legislative actions.

Lessons Learned

The number of performance indicators used in a given system probably should not exceed 20. The goal should be to produce a general institutional assessment rather than a precise measurement. Indicators are simply one of many tools available to monitor quality assurance, enhancement, efficiency, and effectiveness.

State-level or top-down models do not normally produce the constructive, substantive, and sustained results that campus-based and -initiated performance systems produce. Externally imposed systems are rarely accepted by internal stakeholders, such as faculty. The frequent result is resistance rather than reform.

Two central constituencies essential to the success of such models are the faculty and the legislature. Both parties need to be involved in developing the model and need to buy into the processes. Change must be initiated by an institution's administrative leadership.

The performance indicators proposed often lack validity and reliability. This shortcoming is particularly prevalent in top-down, externally imposed systems that have been hastily adopted, using what Ewell (1996) termed "legislation by fax." The public and legislature seem enamored of certain indicators, such as small classes and senior faculty teaching lower-division courses, that are not necessarily effective in promoting learning.

Capstone institutions will be limited somewhat in the qualitative improvement they can demonstrate with certain growth-oriented indicators, such as persistence rates and graduation rates. Rather than a value-added approach, benchmarking with a national sample of institutions, the approach employed by the State University of New York, may offer greater promise for demonstrating efficiency and productivity.

A single indicator model cannot be applied to all types of institutions effectively without diminishing or distorting diverse missions and exacerbating the traditional conflict in priorities among teaching, research, and service. Flexibility is necessary to fit the mission of the institution or agency.

Despite the current campus emphasis on using qualitative vehicles such as portfolios, in the case of state systems, performance indicators should almost always be quantitative in nature, because policymakers are dubious about results that cannot be audited, verified, or submitted to precise comparison. Furthermore, graduate education and agency performance, for example, are rarely incorporated into these models. Improving undergraduate education was paramount during the early development of such indicator systems, particularly top-down models, but the current trend is toward ensuring better use of resources, and linking resources to outcomes.

The indicators should have real financial incentives for institutions and agencies, and direct individual rewards for those directly responsible for success in meeting the goals.

Finally, many of the accountability systems that have been ineffective failed to embrace or communicate in timely, understandable terms the results obtained from the performance indicator model. Innovative approaches such as a home page on the Web, a flier of the sort used by the Colorado System, and an insert for the Sunday edition of the major papers are ways to get an institution's message across.

Project Continuation

The results of the project have been shared with a number of institutions and states. For fiscal year 1999-2000 the state of Texas has added some indicators to its statewide system and particularly focused on a specific set of indicators for athletics. It is unknown how performance indicators will be transformed into funding elements, but the project plans to compare its results to a set of national benchmarks, to gauge its progress on institutional goals in state agency and system planning efforts, and to gauge consumer satisfaction with the educational product.

Available Information

The project has generated three monographs, several articles, a set of published proceedings, and several national and one international presentation. General information about the project may be obtained by contacting the following:

Jerry Gaither
Prairie View A&M University
Institutional Effectiveness, Research, and Analysis
P.O. Box 65
Prairie View, TX 77446
Telephone: 409-857-2448
Fax: 409-857-2451

or

Lee Blank
Texas A&M University System
309 Administration Bldg.
College Station, TX 77843-1259
Telephone: 409-845-6646
Fax: 409-845-6358

or

Verna Dewees
The Texas A&M University System
College Station, TX 77843-1138
Telephone: 409-845-6459
Fax: 409-845-8695

or

Ray Hayes
Texas A&M University - Corpus Christi
6300 Ocean Drive
Corpus Christi, TX 78412
Telephone: 512-994-2321
Fax: 512-994-5810

FIPSE Home