A r c h i v e d  I n f o r m a t i o n

Office of the Under Secretary
Executive Summary

When Schools Stay Open Late:
The National Evaluation of the 21st-Century Community Learning Centers Program

Summary of First Year Findings

In an era when most parents work, many Americans want their children to have access to safe and supervised after-school activities that can help develop academic, personal, and social skills. In 1994, Congress authorized the 21st-Century Community Learning Centers (21st-Century) program to open up schools for broader use by their communities. In 1998, the program was refocused on supporting schools to provide school-based academic and recreational activities after school and during other times when schools were not in regular session, such as on weekends, holidays, and during summers. As an after-school program, 21st-Century grew quickly from an appropriation of $40 million in fiscal year 1998 to $1 billion in fiscal year 2002. It now supports after-school programs in about 7,500 rural and inner-city public schools in more than 1,400 communities. Programs operate in public school buildings and offer academic, recreational, and cultural activities during after-school hours. A distinguishing characteristic of 21st-Century programs is the inclusion of academic activities. Grants made after April 1998 included a requirement that programs include academic activities.

This study, conducted for the U.S. Department of Education with support for additional data collection and analysis from the Charles Stewart Mott Foundation, presents the first-year findings of the largest and most rigorous examination to date of school-based after-school programs.¹ The study was designed to examine the characteristics and outcomes of typical programs and did not attempt to define or identify the characteristics of the best programs. Programs selected to be in the study operated in elementary and middle schools. Some were in their second year of funding when the study began collecting data and others were in their third year of funding. Most grantees that were part of the study had operated some type of after-school program before receiving a 21st-Century grant and were using their grant funds to expand or modify their services and activities. About 65 percent of middle school grantees and about 57 percent of elementary school grantees in the study had operated after-school programs in one or more schools that were part of the 21st-Century grant.

The study currently is collecting another year of follow-up data and has expanded to include more programs serving elementary school students. The additional data from the second follow- up year and from the newly included programs will be the basis for two future reports. The first will update the findings for middle school students using another year of follow-up data and will present first year findings for elementary school students using a larger number of elementary school programs. The second will update the findings for elementary school students using another year of follow-up data.

Key Impact Findings

The first-year findings reveal that while 21st-Century after-school centers changed where and with whom students spent some of their after-school time and increased parental involvement, they had limited influence on academic performance, no influence on feelings of safety or on the number of "latchkey" children and some negative influences on behavior.² In brief, the key findings are:

• Limited Academic Impact. At the elementary school level, reading test scores and grades in most subjects were not higher for program participants than for similar students not attending the program. In addition, on average, programs had no impact on whether students completed their homework or completed assignments to their teacher's satisfaction.

  For middle school students, grades in most subjects were not different than for similar students not attending the 21st-Century program. Grades for math were higher for 21st-Century participants, but the overall difference was small. A subgroup analysis found larger grade point improvements for black and Hispanic middle school students and their teachers also reported less absenteeism and tardiness compared with nonparticipants. Teachers for middle school students were more likely to say assignments were completed to their satisfaction, although program participants were not more likely to do or complete the homework assigned. Another subgroup analysis found that students who attended programs more frequently, both at the middle school and elementary school levels, did not have higher academic outcomes compared with students that attended less frequently. Other analyses did not find statistically significant relationships between program characteristics, including program maturity, and academic impacts.

• Adult Care Increased but Self-Care Unaffected. The findings indicate that programs reduced the proportion of students being cared for by parents and by older siblings, and increased the proportion of students being cared for by non-parent adults. The net effect was to increase the proportion of students being cared for by an adult (either a parent or a non-parent adult), by reducing the proportion being cared for by an older sibling.

  Programs did not reduce the percentage of students in self-care (who are commonly referred to as "latchkey" children). Students were defined to be in self-care if they (or their parents, for elementary school students in grades K-2) indicated that they were not in the presence of adults or older siblings after school (they were by themselves, with others their age, or with younger siblings after school). Other definitions of self-care, such as whether students ever said they were by themselves after school, were analyzed with similar results. The most common care arrangement for nonparticipants was for students to go home after school and be cared for by a parent, which was true for about 53 percent of middle school students in the comparison group and 67 percent of elementary school students in the control group.

• No Improvements in Safety and Behavior. Programs did not increase students' feeling of safety after school. At the middle school level, participants were more likely to report that they had sold drugs "some" or "a lot" and somewhat more likely to report that they smoked marijuana "some" or "a lot" (although the incidence was low). Participants also were more likely to have had their property damaged. (Data on these items were not collected for elementary school students.) No impacts were found on other measures of behavior.

• Increased Parental Involvement. At the middle school level, programs were associated with increased parent involvement at their child's school. Parents of program participants were more likely to volunteer at their child's school and attend open houses or parent-teacher organization meetings. Parents of elementary school level program participants were more likely to help their child with homework or ask about things they were doing in class.

• Negligible Impact on Developmental Outcomes. Programs had no impacts on developmental outcomes, such as whether students felt they were better able to plan, set goals, or work with a team. At the middle school level, program participants were less likely to rate themselves as "good" or "excellent" at working out conflicts with others.

Key Implementation Findings

The first-year findings indicate that grantees generally had succeeded in implementing their planned programs and in gaining support from and creating working relationships with school principals and teachers. Most programs provided academic, enrichment, and recreation activities, with homework help being the most common academic activity. The mix across the three activity areas varied according to locally determined needs and preferences. A few programs focused only on providing academic activities, but none focused only on providing recreational activities. The federal grant and other funding sources enabled programs to spend about $1,000 for each student enrolled during the school year, equivalent to about a 16 percent increase in education spending. Other implementation findings include:

• Low Levels of Student Participation. Attendance in the programs was low, averaging less than two days a week, despite the fact that programs typically were available to participants four to five days a week.

• Programs Staffed Predominantly by School-Day Teachers. A third of the program coordinators and three out of five program staff members were school-day teachers. To accommodate the varying schedules and requirements of teachers, staff members often worked only a few days a week and for short periods.

• Limited Efforts to Form Partnerships and Plan for Sustainability. Programs did not collaborate much with other community organizations. In general, centers contracted with community agencies to provide specific after-school sessions rather than as partners with shared governance or combined operations. Programs also were slow to begin planning to sustain themselves after the 21st-Century grant ends. Even among those grantees within months of their grant's end, sustainability planning was almost nonexistent.

Overall, the findings suggest that policymakers and program developers need to consider ways to address low student participation and low academic content. Considering program structures that would facilitate more frequent attendance, such as focusing on serving students having difficulty in reading or math and asking them to participate a minimum number of days each week, may be worth considering. Efforts to increase the academic content and quality of activities also may be fruitful. Especially for middle school students, the challenge will be how to both attract students and help students improve their academic performance.

Methodology

While research has evaluated other after-school programs, this study--conducted by Mathematica Policy Research, Inc. (MPR) and its partner, Decision Information Resources, Inc.--is one of the few that is consistent with the principles of scientifically based research set out in the recent No Child Left Behind Act. The study is unique in the large number of after-school programs that were included and in its application of rigorous techniques for measuring impacts.

The evaluation's design includes a middle school study and an elementary school study. The middle school study is based on a nationally representative sample of after-school programs and participants and a matched comparison group of students which is similar to the program participant group. Similar students were identified in host schools or in other schools in the participating districts. Thirty-four school districts and 62 centers in the districts are included in the study.

The elementary school study uses random assignment of students to treatment and control groups. The study involved 14 school districts and 34 centers. Results presented here are from seven school districts selected in the first year of the study; another seven school districts were added in the second year of the study and data currently are being collected in these districts. The elementary school programs that were part of the study appear to be typical of elementary school 21st-Century programs along most dimensions (although they tended to be more urban and served a larger percentage of minority students than the average elementary program). However, caution should be exercised in applying the findings to all elementary school programs. Programs in the study had more applicants for their slots than they could serve, which facilitated the use of an experimental design, but the programs were not statistically sampled.

The findings presented in this report are based on one year of data collected in school year 2000-2001 from students, parents, teachers, principals, program staff members, and school records. Evaluators collected baseline and follow-up data for 4,400 middle school students and 1,000 elementary school students, and conducted site visits, lasting between two and four days, to all grantees at least once. MPR is continuing to study the programs and will prepare two additional reports based on another year of follow-up data and another round of visits to each program.

General Information about 21st-Century Programs

Annual performance reports submitted by grantees to the U.S. Department of Education indicate general characteristics and context of 21st-Century programs. The reports also are informative about centers in the study. Nationwide, the average grantee ran three or four centers that together reported enrollment of almost 700 students over the course of the school year. Attendance varied by day, with some students attending regularly and others more occasionally, and with students enrolling and exiting from the program at different points during the year. Fifty-seven percent were minority students, compared with 37 percent of students nationwide. Most centers (95 percent) were located in elementary or middle schools or located in schools that included some combination of K-8. Typically, centers were open 10 or more hours a week, after school, and a third were open 20 hours or more a week. Some were open on Saturdays, and many offered summer programs. Sixty-six percent of host schools were considered high-poverty (at least half their students were eligible for free or reduced-price lunches). Nationally, 17 percent of schools are high-poverty. Center budgets averaged about $196,000 a center, or about $1,000 per enrolled student, with the 21st-Century grant accounting for about 70 percent of budgets. Programs typically were free both for students and parents.

The rest of this summary looks at findings for middle school programs, then at findings for elementary school programs. These findings are based on the various samples that were drawn by this study. We present the findings for middle and elementary schools separately because of differences in how the programs were selected for the evaluation and how impacts were measured.

Findings for Middle School Programs

Middle school centers in the study usually offered the following activities:

• Academic help, primarily supervised daily homework sessions. Nearly 9 of 10 middle school centers (89 percent) provided homework help. Slightly more than half (54 percent) provided homework help and other academic support, such as tutoring, state test preparation, and help sessions in reading, writing, and math skills. Help sessions usually were scheduled between one and three days a week, staffed by certified teachers, and targeted to particular students, such as those referred by a classroom teacher or those performing poorly on state standardized tests.

  In spite of the focus on homework support, fewer than two in five students (38 percent) said that the centers were a good place to get homework done. Consistent with this finding, site visitors observed that homework sessions usually were organized with students in large groups proctored by teachers or other staff members, with students talking to each other and staff members not checking the homework for quality or completeness.

• Recreation activities, such as using the gym, playing board games, or using computers. These often were part of the daily student fare although content varied according to the day.

• Cultural and interpersonal enrichment, including crafts, drama, music, mentoring, role modeling and conflict resolution, and issue forums. These activities were offered most days of the week but not necessarily every day. Specific activities might occur just once or twice during the week.

Management and Staffing

Officials from the host school or district oversaw most middle school programs. Program directors usually had supervisory and administrative roles, while program coordinators handled day-to-day details of the centers, such as recruitment, scheduling, staffing, parent and community outreach, and attendance monitoring. Nearly all other staff members were directly involved in student activities or instruction and spent most of their time working with students. Survey data showed that middle school teachers believed that, as a result of working with students at the centers, they improved their teaching skills and had better relationships with some students.

Student Participation

Middle school students in the study attended centers for 32 days-- about one day a week--during the 2000-2001 school year. More than half attended for fewer than 25 days, a quarter attended for more than 50 days, and almost 10 percent attended for more than 75 days (see box). Program staff attributed the low attendance to the lack of interesting or appealing activities and to competition from other organized activities, especially sports. Center policies also made it easy not to attend--many allowed students to participate on a drop-in basis, choosing each day whether or not to participate.

Not all students chose to participate in 21st-Century programs. Students who had chosen not to participate (surveyed in six selected programs) said that they would rather "hang out" after school, were involved in other organized activities after school, or were not interested in the activities. Almost half of the students thought the centers were "mostly a place kids go when their parents are at work," and a quarter considered them "just for kids who need help in school." Participants who had stopped attending echoed these sentiments.

Learning Outcomes

The objective of improving learning outcomes distinguished 21st-Century after-school programs, and more than 75 percent of parents of participants said they believed participation would help their child do better in school. However, participants were just as likely as comparison group students to complete homework, although they were more likely to do so to their teachers' satisfaction, and participants had about the same English, science, and social studies or history grades as similar students. Participants had slightly higher math grades (see box below), and slightly higher school attendance.

Additional Analyses and Other Outcomes

The evidence on the effect of programs on student effort in school is mixed. According to teachers, program students were more likely than similar students to try hard in reading or English class, be attentive in class, and participate and volunteer in class. However, teachers also report similar rates of frequent homework completion for program participant and nonparticipants. In addition, program participants report spending a similar number of hours watching TV.

Another program objective was to reduce students' exposure to unsafe settings. However, programs did not increase the extent to which students felt safer after school, and, although rates were not high, participants were more likely to report that they sold drugs, smoked marijuana, and, especially for girls, had their personal property damaged or were "picked on." Other measures of behavior--such as suspensions, absences, and teacher reports of discipline problems--were the same in both groups.

In general, program participation did not change students' interpersonal skills. Program students were no more likely to report getting along with others their age, feeling included, being good at working with others in a team, or setting a goal and working to achieve it. In fact, middle school participants were less likely their nonparticipant peers to rate themselves as good or excellent at working out conflicts with others.

NOTE: Reported impacts were estimated using regression models to adjust for baseline differences between program participants and the similar students. The adjustment variables in the regression models included student demographic characteristics, household socioeconomic status, and students' baseline test scores, attendance, disciplinary problems, and self-reported grades.

Impacts by program characteristics were also estimated. These analyses focused on two types of program characteristics: (1) program emphasis on academics and (2) levels of participant attendance. Interestingly, programs that emphasized academic activities over recreation and other activities were not more likely to increase test scores or grades. Similarly, no relationship is evident between average attendance of a program and impacts by program.

Additional analysis looked at the impacts for frequent participants compared to infrequent participants. The analysis suggests that frequent participants were more likely to be from disadvantaged households and to want to improve in school, as their better behavior in school and their more frequent attendance itself indicate. However, the analysis did not reveal that more frequent participation led to better outcomes.

Findings for Elementary School Programs

Researchers selected elementary school centers that had more applicants than they could accept, because these centers could implement experimental designs. Elementary school programs in the study were more likely to be in urban areas and to serve more disadvantaged students than other elementary school programs, but most characteristics were similar to other elementary school programs.

Elementary school students attended for 58 days, on average, during the school year, and more than one-third of students attended for more than 75 days. These attendance levels may not be typical of attendance levels of elementary school programs in general because the evaluation looked only at oversubscribed programs.

The elementary school programs in the study increased the time students spent at school or outside the home and reduced the time spent at home after school cared for by a parent or sibling. Programs did not reduce self-care, the incidence of which was low (about two percent of students).

The programs had no effects on reading or math grades or reading test scores. For example, in spring 2001, program students had an average percentile reading score of 34.3, compared with a score of 34.1 for similar students. Social studies grades were higher by a statistically significant margin (83, compared with 80), but grades in other subjects were not.

Programs did not appear to improve student effort in school. Parents and teachers had different views about whether effort improved. According to teachers, program students were more likely than similar students to try hard in reading or English class. According to parents, however, program students were less likely than similar students to work hard in school. However, students reported no differences in homework completion, time spent watching television, or time spent reading for fun.

NOTE: Impacts were estimated using regression models to adjust for differences between treatment group and control groups in fall 2000. The adjustment variables in the regression included indicators of students' demographic characteristics, household socioeconomic status, and students' fall test scores, as well as previous year attendance, disciplinary problems, and self-reported grades.

Programs did not affect whether students felt safe or unsafe after school and did not affect student behavior in school. Suspensions, absences, and teacher reports of discipline problems were the same for both groups.

Program participation did not change students' interpersonal skills. Program students were no more likely to report getting along with others their age, feeling included, being good at working with others in a team, or setting a goal and working to achieve it.

Directions for the Future

These findings reflect the challenges school-based after-school programs face to improve student outcomes. Even for after-school programs oriented toward providing academic support as well as recreational and social activities, there were few improvements in homework completion, grades, and test scores. The lack of academic improvement may be due to the low attendance rates and the length of the follow-up period. However, analyses of those who participate more frequently found that more attendance alone may not make measurable differences in outcomes. In addition, too few participants may have received sustained, substantive academic support. Both participation rates and the content of program academic offerings may need more attention.

The No Child Left Behind Act restructures the 21st-Century program and focuses more attention on the program's potential for improving academic outcomes, especially for disadvantaged students. An additional year of follow-up and the expansion of the number of elementary school programs in the study will provide another opportunity to assess whether the programs (as they are currently implemented) are likely to meet these objectives.

¹ This study focuses on school-based programs that are part of the 21st-Century program. Results do not extrapolate to all after-school programs in general.

² A "center" refers to after-school services operated in one school, and a "program" refers to one or more centers operated in one school district. The study measured impacts at the program level but not at the center level.

[Return to Contents]
This page last modified--December 31, 2002 (jer)