The Costs and Effectiveness of Educational Technology - November 1995

---

Data on the benefits of optimal school-wide technology implementations, especially implementations in the service of school reform that aims at new student outcomes, new approaches for assessing student outcomes, and new instructional strategies (e.g., a significant measure of individualized student learning using a many-pathed, project-based curriculum) are not and will not soon be available. James Kulik, Bill Hadley, Dexter Fletcher and Luis Osin provided different but overlapping slants on what we know from experimental and empirical data about the effectiveness of student learning using computer technology for the case of limited and well defined curriculum objectives. (We discuss later in the section titled "evaluation" what can be inferred from this knowledge.)

James Kulik (University of Michigan)

Kulik's presentation was based on his recent article,¹⁶ which opened with, "What do evaluation studies say about computer-based instruction? It is not easy to give a simple answer to the question. The term computer-based instruction has been applied to too many different programs, and the term evaluation has been used in too many different ways." He goes on to describe the meta-analytic approach for creating a composite picture of findings on computer-based instruction, and presents an overview of these findings. We quote: "At least a dozen meta-analyses have been carried out to answer questions about the effectiveness of computer-based instruction (Table 1.1, in original). The analyses were conducted independently by research teams at eight different research centers. The research teams focused on different uses of the computer with different populations, and they also differed in the methods they used to find studies and analyze study results. Nonetheless, each of the analyses yielded the conclusion that programs of computer-based instruction have a positive record in the evaluation literature."

"The following are some major points emerging from these meta-analyses:

• Students usually learn more in classes in which they receive computer-based instruction. The analyses produce slightly different estimates of the magnitude of the computer effect, but all the estimates were positive. At the low end of the estimates was an average effect size¹⁷ of 0.22 in 18 studies conducted in elementary and high school science courses (Willett, Yamashita & Anderson, 1983). At the other end of the scale, Schmidt, Weinstein, Niemiec, and Walbert (1985) found an average effect size 0.57 in 18 studies conducted in special education classes. The weighted average effect size in the 12 meta-analyses was 0.35¹⁸. This means the average effect of computer-based instruction was to raise examination scores by 0.35 standard deviations, or from the 50th to the 64th percentile.

• Students learn their lessons in less time with computer-based instruction. The average reduction in instructional time was 34% in 17 studies of college instruction, and 24% in 15 studies of adult education (C.-L. C. Kulik & J. A. Kulik, 1991).

• Students also like their classes more when they receive computer help in them. The average effect of computer-based instruction in 22 studies was to raise attitude-toward-instruction scores by 0.28 standard deviations (C.-L. C. Kulik & J. A. Kulik, 1991).

"This brief review shows that there is a good deal of agreement among meta-analysts on the basic facts about computer-based instruction. All the meta-analyses that I have been able to locate show that adding computer-based instruction to a school program, on the average, improves the results of the program. But the meta-analyses differ somewhat on the size of the gains to be expected. We need to look more closely at the studies to determine which factors might cause variation in meta-analytic results."

Kulik goes on to examine more closely a set of 97 evaluations of computer-based instruction in an attempt to reach more precise conclusions about their effectiveness, and concludes: "Meta-analysts have demonstrated repeatedly that programs of computer-based instruction usually have positive effects on student learning. This conclusion has emerged from too many separate meta-analyses to be considered controversial. Nonetheless, results are not the same in every study of computer-based instruction. No meta-analyst has reported that all types of computer-based instruction increase student achievement in all types of settings. Study results are not that consistent, nor would we want them to be. Computer-based instruction is a loose category of innovation. It covers some practices that usually work and other programs that have little to offer.

"Breaking studies of computer-based instruction into conventional categories clarifies the evaluation results. One kind of computer application that usually produces positive results in elementary and high school classes is computer tutoring.¹⁹ Students usually learn more in classes that include computer tutoring. On the other hand, precollege results are unimpressive for several other computer applications: managing,²⁰ simulations, enrichment, and programming."

"The overall findings on computer tutoring compare favorably with findings on other innovations. Few innovations in precollege teaching have effects as large as those of computer tutorials. Effects are especially large and consistent in well designed programs such as the Stanford-CCC program. Programs of curricular change that provide more challenge for high-aptitude students may have produced more dramatic effects in evaluation studies, but such programs affect only a limited part of the school population. The effects of computer tutoring are as great as those of peer- and cross-age tutoring, and they are clearly greater than the gains produced by instructional technologies that rely on print materials."

Bill Hadley (Langley High School)

Hadley, a teacher on a year's leave at Carnegie-Mellon University (CMU) from Langley High School, Pittsburgh, PA, reported on the Pittsburgh Urban Mathematics Project (PUMP). Based on a curriculum that emphasizes multiple representations (words, symbols, graphs, etc.), the program partially depends for its delivery on software--a so-called intelligent tutor--designed and developed by John R. Anderson, Prof. of Psychology and Computer Science at CMU. PUMP has been operational at Langley H.S. for three years.

With many of its students eschewing algebra in favor of a general mathematics track, which practically proscribes the necessary achievement in mathematics and science education for a successful technical career of any sort, Langley sought a solution in the combination of a revised algebra curriculum²¹ and Anderson's algebra tutor,²² which was developed over a period of years with National Science Foundation R&D support. A statement of Langley's main objectives is:

• to have all students be successful in first year algebra and geometry;

• to increase the number of students in higher mathematics classes;

• to have all students make conceptual and practical connections between algebra and the world outside of school; and,

• to prepare students for the "world of work" as well as further academic study in mathematics.

Students spend five periods a week in algebra study, two with the intelligent tutor in a laboratory setting and three in a regular classroom. Carrying out exercises, which is greatly facilitated by the computer laboratory, is a primary element of an instructional strategy aimed at improving student learning.

As a result of a satisfyingly successful experimental first year in which 73% of the students enrolled in the PUMP first-year algebra course passed, while 56% of students enrolled in the regular algebra course failed, the PUMP first-year algebra course was made a required course of study for all Langley students. In the second year of the adoption, 61 of the 73 students passing the PUMP first-year algebra course enrolled in geometry; and of those 37 enrolled in Algebra 2 in year three. By contrast, 20 of the 24 students passing the regular algebra course during the experimental first year went on to take geometry; and of those only three enrolled in Algebra two. Two other schools in the Pittsburgh area recently adopted PUMP, despite the financial investment in hardware this step requires, but have not yet made first-year algebra a required course for all students.

J. Dexter Fletcher (Institute for Defense Analyses)

Fletcher, speaking to the use of technology in military training, started by remarking on a number of features distinguishing military training from K-12 education:

• Military training involves bringing individuals or collections of individuals to a required level of performance in the conduct of prescribed tasks. The time to reach this level of competence, a variable among trainees, affects training cost in several ways, and therefore cost-effectiveness, which sharply affects the selected training approach.

• The costs of training (most of which are time dependent) that employs "hard" technology can be roughly divided into three categories: the trainee cost, the hard technology cost, and an overhead cost. The trainee is paid while in training; there is an added cost if the trainee is removed from the field and a replacement is required; and travel and living expenses for the trainee and any replacement may constitute yet another cost. The costs for hard technology, say a flight simulator, include R&D, and production and maintenance, which have to be amortized over the lifetime of the equipment, and contribute a time-dependent amount to the cost of the individual's training. Overhead costs include everything else, like maintenance of the training site, and so on, which also contribute a time-dependent amount to the total cost of the individual's training. Time to train is therefore a major consideration for DoD.

• Unlike the case for schools, the locus of decision-making for the use of technology in military training is not local and decentralized, but centralized and even highly centralized where initial R&D support is required.

• In summary, the DoD uses technology to regain the benefit of individual tutoring that has been lost to the economic necessity of training students in large classes. Bloom²³ writes that the difference between individualized tutoring and group instruction may account for as much as two standard deviations in measured achievement.

Effectiveness. Fletcher then went on to discuss the issues of effectiveness, cost and cost-effectiveness in turn, pointing out that the available data for considering these training criteria diminish in that order. Emphasizing the importance of individualization (for pace, difficulty, content, sequence and style) to effectiveness, he presented data on effect sizes for training using interactive (computer-controlled) videodisc instruction compared with more conventional approaches like platform lecture, textbooks, workbooks, and the use of actual equipment for practice. Videodisc functionalities in these comparisons give some indication of the effectiveness of multimedia approaches to instruction. The average effect size across 47 evaluations of interactive videodisc instruction used in military training, industrial training, and higher education was 0.50, or an increase from 50th to 69th percentile levels of performance. Considering the three settings separately, the effect size was 0.39 for military training (an increase from 50th to 65th percentile performance), 0.51 for industrial training (an increase from 50th to 70th percentile performance), and 0.69 for higher education (an increase from 50th to 75th percentile performance).

Little difference in effectiveness is found between knowledge and skill performance measures in interactive videodisc instruction evaluations; i.e., interactive videodisc instruction appeared to be equally effective for both. Studies that compared the interactive intensity of interactive videodisc instruction found significantly better results for the more interactively intense applications.

The average effect size in 38 studies of military training using computer-based instruction (CBI) is reported as 0.40 for experimental groups numbering less than 20, in the same range as effect size in education reported by Kulik; and 0.30 averaged over 35 studies for experimental groups numbering greater than 20.

Effect size has also been used to assess simulation applications for training to maintain and repair devices, that is, the use of simulated equipment is compared to the use of actual equipment, with training time held constant and success in maintaining or repairing actual equipment used as the final performance measure. Average effect size in these studies has been found to be 0.40 (an increase from 50th to 66th percentile performance). And most notably, the cost of training using simulated equipment was found to be about a third the cost of training using actual equipment.

Evaluations of interactive videodisc used to simulate actual equipment have been performed using two approaches: videodisc used only in simulator mode and videodisc used in both simulation mode and for tutorial guidance during the simulation. Effect sizes for these two approaches were 0.14 (an increase from 50th to 56th percentile performance) and 0.41 (an increase from 50th to 66th percentile performance), respectively. This finding is consistent with others that have found an interaction effect between the sophistication of the students and the amount of tutorial guidance needed in simulation-based training. "Naive" students benefit from tutorial assistance.

Finally, Fletcher reported on a group of assessments of computer-based instruction in K-12 education, which predicted performance on standardized achievement tests solely on the basis of the amount of time each student spent to complete a preparatory program of computer-based instruction. Scores on the comprehensive tests of mathematics achievement could be predicted to the nearest tenth of a grade-placement using these time measures exclusively. Faster learners are better learners.

Cost. In military training, the time required to train to a required level of performance sharply affects almost all cost elements contributing to total cost. Repeated analyses have found that, on average, technology reduces the time to reach criterion levels of knowledge and performance by about 30 percent, in the same range of reduced instructional time in education reported by Kulik.

Cost Effectiveness. For the case of military training by simulation, which can never entirely replace the "real thing",²⁴ assessing the cost-effectiveness of simulation, whether by general-purpose hardware or special equipment, (and setting aside any other costs,) requires consideration of an additional factor, the transfer effectiveness ratio (TER). For a flight simulator, for example, this would be computed as the difference between actual aircraft time without simulator training and aircraft time with simulator training (each of which has a cost) divided by simulator time.

That is, up to the limit for which the simulator can realistically simulate general air work, an hour of simulator time saves an hour of training time in an actual aircaft. For this case, if the cost of an hour of simulator training is less than the cost of an aircraft hour, simulator time is cost-effctive.

For K-12 education, Fletcher summarized his paper²⁵ comparing the costs (in constant 1985 dollars) to increase comprehensive mathematics scores (computation, concepts and word problems) one standard deviation using different approaches: tutors, reduced class size, increased instructional time, and providing computer based instruction. The results appear below.

Approach
Tutoring (20 min/day)
Cost for 1 sd gain ($) by peers
286. by adults 1612.
Reduced class size 35 to 30
983. 30 to 25
1171. 25 to 20
1367. 35 to 20
1195. Increased instructional time 30 min/day
2667. CBI (10 min/day 3rd grade and 11/2 min/day 5th grade))
grade 3 (computation) 338. grade 3 (concepts)
208. grade 3 (word problems)
192. grade 5 (computation)
462. grade 5 (concepts)
490. grade 5 (word problems) 206.

Peer tutoring and CBI are revealed by these data to be the most cost-effective approaches. Fletcher observed these two approaches could be combined and reconciled by having two or three students clustered together at a single CBI terminal.

Summing up the military experience, Fletcher offered the following principles to guide the use of technology for training:

• for practice rather than initial learning;

• to simulate expensive equipment or dangerous field conditions;

• to provide self-study for remote or dispersed learners; and,

• to closely monitor progress in student learning.

Concerning the conceptualization and practice of assessment, he suggested assessments should:

• be designed to inform specific choices;

• address both formative (design issues) and summative issues;

• expand the range of instructional outcomes considered and support the development of principles of instructional design that emphasize specifiable outcomes;

• validate instructional objectives to assure not only improved instruction, but improved instructional objectives;

• consider broad ranges of instructional inputs and outcomes, rather than narrow ones;

• consider group performance; and,

• consider cost.

Luis Osin (Centre for Educational Technology, Israel)

Luis Osin, on leave at the Learning Research & Development Center of the Univ. of Pittsburgh from the Centre for Educational Technology (CET) in Israel, reported on the CET computers in instruction program in Israel, where almost 700 schools use CET computer systems comprising some 18,000 student stations utilized by nearly 180,000 students annually. With the goal of adapting instruction to the individual learner--the learner's current knowledge, cognitive learning style and pace, CET offers a full range of services, including advice to schools and education authorities, supply and installation of computerized systems, in-school teacher training, and full system maintenance of hardware, software and courseware (application software). Indeed, almost 30 communities with some 150 schools have elected to operate under the direct supervision of CET, while hundreds of additional schools have adopted some of its adaptive teaching and learning methods.

Starting with the observation that individualized instruction is necessary to overcome the well-known distribution of the age-grade of students at any grade level, (e.g., nominally 4th grade students typically range in achievement between 2nd and 6th grades; nominally 6th grade students typically range in achievement between the 3rd and 9th grades; and so on,) Osin went on to describe the role of computers in CET's adaptive teaching and learning strategy:

• Individualized dialog with every student. By interacting with the computer, every student may learn according to his/her cognitive level and learning speed, independently of the cognitive styles and learning pace of the student's classmates. The student is able to hold a "conversation" with the (software) author, and receive explanations matched to his/her level of learning.

• Tools for information processing. Teachers and students may enjoy using general purpose (feature rich) tools like text and graphic editors, spreadsheets, etc. The teaching and learning of many subjects can be based, at least partially, on the utilization of these tools.

• Access to remote information. Today, it is possible to access large and updated databanks, located not only in one's own country, but dispersed around the world.

• Communication with others around the world. Students in different schools, cities and countries can cooperate on a common project. Teachers can benefit from advice and support of remote educational R&D centers.

• Stimulating presentation. Material can be presented to the student with all of the expressive possibilities of modern cinematography.

A well-rationalized program includes intensive teacher preparation and the engineering design of content software (i.e., many cycles of trial and improvement), which may be more top-down than would be acceptable in U.S. schools. The result was revealed in one example of a school with a low SES population in which an exceptionally and unusually high percentage of the entire student body was performing on or above grade level in mathematics achievement.

Osin observed that the principal beneficiaries of individualized instruction were slow or low-aptitude learners, and that computer resources could be allocated in such a way as to prevent them from falling behind their age cohort, with the usual educationally destructive consequences.

---

¹⁶ Meta-Analytic Studies of Findings on Computer-Based Instruction by James A. Kulik, 1994, in Technology Assessment in Education and Training, E.L. Baker and H.F. O'Neil, Jr., (eds.), Hillsdale, NJ: Lawrence Erlbaum.

¹⁷ The meaning of effect size used here is the so-called standardized mean difference. This index gives the number of standard deviation units that separates outcome scores of experimental and control groups. It is calculated by subtracting the average score of the control group from the average score of the experimental group and dividing the remainder by the standard deviation of the measure. From Kulik, for example, if a group that receives computer-based coaching on the Scholastic Aptitude Test (SAT) obtains an average score of 550 on the test, whereas a group that receives conventional teaching averages 500, the effect size for the coaching treatment is 0.5, because the standard deviation on the SAT is 100.

¹⁸ An effect size of 0.32 can be thought of as equivalent to a gain of about three months on an age-grade equivalent scale.

¹⁹ Kulik defines computer tutoring as a program in which the computer presents material, evaluates responses, determines what to present next, and keeps records of progress. Drill-and-practice software belongs in this category.

²⁰ As in computer-managed instruction.

²¹ Developed primarily by Bill Hadley.

²² The computer equipment for a laboratory implementation of the program was donated by the Apple Computer, Inc.

²³ B. S. Bloom, The Two Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring," Educational Researcher, 13, 1984, pp. 4-16.

²⁴ K-12 education does not much consider this, which would give new meaning to the stated goal of preparing students for the world of work.

²⁵ J. D. Fletcher, D. E. Hawley, and P. K. Piele, "Costs, Effects and Utility of Microcomputer Assisted Instruction in the Classroom," American Educational Research Journal, 27, 1990, pp. 783-806.
-###-

---

Previous	Table of Contents	Next