[ Appendix C: Exhibit C-2 ]
[ Regression Analysis Result Summary Tables:
Exhibits C-3, C-4, and C-5 ]
Appendix C: Multivariate Analyses in Chapters 5 and 6
The rationale for the Even Start program rests upon general postulates or expectations that regular participation in family-focused educational services of sufficient intensity and quality will promote improvement of literacy and basic skills of parents; general development and school readiness of children; and self-sufficiency of the family as a whole. The Even Start statute reflects these expectations by specifying many aspects of the program to be implemented: regular, sufficiently intense educational services, support services, home-based instructions, parent-child activities, accommodation to parents? schedules, staff development, local evaluation, interagency collaboration, and integration across three core service areas.
A key mandate for the national evaluation is to investigate whether and to what extent these factors are related to program outcomes. The second national evaluation did not collect measures of program outcomes, except for new families in the Sample Study projects. However, we focused on families? participation patterns as intervening "outcome" measures. The logic behind this strategy was simple: without sufficient levels of participation, Even Start services cannot directly influence educational outcomes of families. In addition to the expectation that different levels and approaches of service delivery may influence the participation measures, there is also a need to know whether the extent of their influence may be moderated by a host of participant characteristics (e.g., participant age, educational background, English proficiency, level of family need) and project characteristics (e.g., program budget, size of project, number of staff, staff qualifications, community setting).
Currently there is insufficient knowledge to formulate specific hypotheses about the potential influence of all these participant, project, and service delivery characteristics on participation patterns. Based on the general assumptions about the Even Start approach, one may construct a regression model that tests a specific hypothesis about characteristics of educational services and the level of educational outcomes. But, the dynamics of factors that influence participation patterns, as opposed to program outcomes, are less known. Therefore, the multiple regression analyses reported in Chapters 5 and 6 are descriptive and exploratory.
We chose to enter variables that represent many of the factors expected to be relevant to families? participation patterns and/or factors that may moderate the relationships between key input factors and participation behaviors. Factors such as parents? pre-Even Start educational attainment, English language proficiency, indicators of family need, or project staff professional qualifications may be considered as moderating variables. On the other hand, some factors (e.g., extent of staff training, home-based versus center-based services, and integration across core service areas) could be viewed as explanatory/causal or moderating factors depending on the version of the Even Start model one holds. The analyses reported in Chapters 5 and 6 left this type of specification open. We simply report the extent of relationship between any of the independent variables with the dependent variable controlling for the effects of all other variables.
In addition to the scarcity of knowledge about factors that affect participation, our analysis design reflects the nature of the available data. The Even Start national evaluation data may be described as "a mile wide and an inch deep." The data cover many aspects of participants, program organizational contexts, implementation approaches, educational contents and intensity, participation patterns, a few progress indicators, and sample-based educational outcome data. In the interest of capturing at least some information about many aspects of the program and at the same time minimizing local projects? data collection burden, no one issue or topic was measured in depth in this evaluation.
Thus, the primary goal of the regression analysis reported in Chapters 5 and 6 is a description of the relationship of each independent variable with a dependent variable holding constant the potential effects of all other variables entered in the analysis. The goal is not prediction of participation levels based on specific participant or project characteristics, or revelation of theoretical relationships among a large array of measures pertinent to Even Start. The descriptive information is intended to support a more focused analysis of variables that reveal relatively strong relationships (primarily by using analysis of variance).
Analytical options available in regression analysis offer many ways to investigate relationships among a set of variables, for example, by combining and recombining independent variables in different sets, creating aggregate variables, transforming the values of independent variables, using many different statistical methods to accommodate different characteristics of data (simple regression, probit, logit, etc.), changing the order in which independent variables are entered in the analyses, etc. Sensible restraint over the analytical activities is needed to avoid losing sight of the primary goal for the analysis and the practical constraints of available time and resources.
Potential problems associated with regression are: (1) multicollinearity (high correlations among independent variables); (2) statistically significant results for individual independent variables by chance from using many independent variables and running many tests; and (3) erroneously raising the R2 (total variance of the dependent variable explained by the multiple regression) purely due to a large number of independent variables.
For the regressions reported in Chapters 5 and 6, multicollinearity is not a problem. The simple, bivariate correlations between any two variables (used as dependent or independent variables in these analyses) are generally very low (Exhibit C.2). The low correlations also did not support combining variables into sets or higher-order measures through factor or cluster analyses.
The risk of finding significant results by chance is minimized in these analyses given the large number of records which essentially constitute the universe of Even Start families, parents, and children. In fact, the large N tends to make even a fairly small regression coefficient statistically significant. However, from each regression analysis, we selected only a few (generally three to five) independent variables that produced the strongest relationships for further, more focused analyses.
The possibility of making erroneous conclusions due to artificially inflated R2s is minimal in our analyses. First, the adjusted ("shrunken") R2s generated by SAS to statistically correct for this problem were essentially the same as the unadjusted R2s, especially for the participant-level analyses based on large Ns. A more important protection against the problem of artificially inflated R2s rests in how we used the regression results. The obtained R2s were relatively low; one (for the number of home visits) reached .22, but many were around .10 or lower. A claim can be made that these R2s are not too low to be useful for analyses based on cross-sectional (versus longitudinal) individual-based data. Nevertheless, these R2s leave most of the variation in the dependent variables unexplained. Thus, we did not stress the explanatory importance of any independent variable or collection of independent variables solely based upon these regression analyses. Even if they had been artificially inflated, the R2s still left much unexplained.
Instead, we used the regression analyses to identify variables that appear to have stronger relationships with the dependent variables compared to other independent variables. We followed up the regression findings with further, more focused analyses of those variables, primarily based on analysis of variance (ANOVA) techniques.
The generally low simple correlations provide perhaps the most reliable conclusion one can derive from all regression analyses conducted for Chapters 5 and 6. The data available can explain only a small portion of variation in these dependent variables (generally about 10 percent).
The remainder of this appendix presents the final summary statistics of stepwise regressions on the following service intensity measures discussed in Chapter 5 and participation measures discussed in Chapter 6:
-###-
[ Appendix C: Exhibit C-2 ] | |
[ Regression Analysis Result Summary Tables: Exhibits C-3, C-4, and C-5 ] |