This report consists of four parts: (1) a description of the Milwaukee Parental Choice Program and the data being collected; (2) a description of the choice families and students; (3) an update on the choice schools; (4) a four-year report on outcomes; and 5) a brief conclusion. Recommendations are appended.
For those readers who have followed our earlier reports, sections one and two will appear familiar. There have been only modest changes to the program, and the characteristics of choice families and students remain consistent over the three years. Because the third year report contained considerable information on the range of choice schools, and because no new schools have been added to the program this year, the third section briefly updates personnel data on the schools. The outcome section focuses on long-term outcomes over four years. Extended information is provided on attrition from the program and on students who applied to the Choice Program in one of the years but were not selected in the random selection process.
In an effort to focus on critical data we have moved several tables in previous reports to the appendices. Several other nonessential tables in prior reports have been excluded either to focus the report more appropriately or because of cutbacks in resources which forced reduction in the data collection and/or analysis effort. Because the amount of information presented in this report is large, we advise readers to carefully review all tables in the text and appendices. Several commentators on previous reports have erroneously stated that certain data were omitted which were in the reports.
II. THE MILWAUKEE PARENTAL CHOICE PROGRAM
The Program. The Milwaukee Parental Choice Program, enacted in spring 1990, provides an opportunity for students meeting specific criteria to attend private, nonsectarian schools in Milwaukee. The program specifications are shown in Figure 1. A payment from public funds equivalent to the Milwaukee Public School (MPS) per-member state-aid (estimated to be $3,209 in 1994-95) is paid to the private schools in lieu of tuition for the student. Students must come from families with incomes not exceeding 1.75 times the national poverty line. New choice students must not have been in private schools in the prior year or in public schools in districts other than MPS. The total number of choice students in any year was limited to 1% of the MPS membership in the first four years, but was increased to 1.5% beginning with the 1994-95 school year.
Schools initially had to limit choice students to 49% of their total enrollment. The legislature increased that to 65% beginning this year. Schools must also admit choice students without discrimination (as specified in s. 118.13, Wisconsin Stats.). Both the statute and administrative rules specify that pupils must be "accepted on a random basis." This has been interpreted to mean that if a school was oversubscribed in a grade, random selection is required in that grade. In addition, in situations in which one child from a family was admitted to the program, a sibling is exempt from random selection even if random selection is required in the child's grade.
The program was designed to prevent payment of public monies to subsidize existing private school students, non-poor families, and students attending religious schools. The Milwaukee program, and the resulting research, cannot be generalized to broader "voucher" programs which would subsidize all private school education. The spirit and the letter of the Milwaukee Parental Choice Program lie in stark contrast to public subsidy of elite or exclusive private education. Rather, the intent is to provide alternative educational opportunities for families that cannot easily exercise choice by residential selection or by purchasing private education.
The program was challenged immediately upon enactment as violating the Wisconsin Constitution. The circuit court denied those challenges in August 1990, and also exempted the private schools from complying with the Wisconsin All Handicapped Children Act. This means that the private schools need not admit learning disabled or emotionally disabled students. The circuit court ruling was overturned by the appeals court in November 1991, but on a 4 to 3 decision, the Wisconsin Supreme Court upheld the constitutionality of the statute in March 1992. This report does not discuss the legal issues.
Research and Data. The study on which this report is based employs a number of methodological approaches. Surveys were mailed in the fall of each year from 1990 to 1994 to all parents who applied for enrollment in one of the choice schools. Similar surveys were sent in May and June of 1991 to a random sample of 5,474 parents of students in the Milwaukee Public Schools. Among other purposes, the surveys were intended to assess parent knowledge of and evaluation of the Choice Program, educational experiences in prior public schools, the extent of parental involvement in prior MPS schools, and the importance of education and the expectations parents hold for their children. We also obtained demographic information on family members. A follow-up survey of choice parents assessing attitudes relating to their year in private schools was mailed in June of each year.1
In addition, detailed case studies were completed in April 1991 in the four private schools that enrolled the majority of the choice students. An additional study was completed in 1992, and six more case studies in the spring of 1993. Case studies of the K-8 schools involved approximately 30 person-days in the schools, including 56 hours of classroom observation and interviews with nearly all of the teachers and administrators in the schools. Smaller schools required less time. Researchers also attended and observed parent and community group meetings, and Board of Director meetings for several schools. Finally, the research includes analysis of three years of outcome measures including data on achievement test scores, attendance, parental attitudes, parental involvement, and attrition from the program.
Finally, beginning in the fall of 1992 and continuing this year, brief mail and phone surveys were completed with as many parents as we could find who chose not to have their children continue in the program.
In accordance with normal research protocol, and with agreement of the private schools, to maintain student confidentiality, reported results are aggregated and schools are not individually identified. Thus these findings should not be construed as an audit or an assessment of the effectiveness of the educational environment in any specific school.
For readers of earlier reports, there may be slight differences in a few of the statistics reported for earlier years. These differences are attributable to improvements in collecting data (such as search for student names in MPS records), errors in reporting, additional survey results that come in after the reports are prepared, and data entry and analysis errors that are subsequently corrected. Nearly all of these differences are very small. Any significant changes are noted in footnotes.
III. CHOICE FAMILIES AND STUDENTS
Numerous questions arise concerning constrained voucher programs and the data on the Milwaukee Choice Program are beginning to answer these questions. How many families apply? How many seats are available in the private schools? How do families learn about the program? Why do they want to participate? What are the demographic characteristics of students and families? What are parental attitudes toward education? What are the prior experiences of applicants in public schools? How well were the students doing in their prior public schools?
Findings are quite consistent over the first four years. For economy, and because four years of data provide a better picture than a single year, most tables contain combined data from 1990 to 1993. Because of the magnitude of the survey response data, much of the descriptive data have been moved to Appendix Tables A1 to A6. Data in those tables report on seven subsamples from the fall, wave one surveys (including the latest sample of Choice applicants for 1993), and Choice and non-selected Choice student responses from wave two, spring surveys. The body of the report contains additive scale score means, standard deviations, scale reliabilities (Table 6a), and significance levels of differences in means of these scales (Table 6b). The most appropriate comparison group to the choice families on most measures is the low-income MPS sample. That group, which includes about two-thirds of Milwaukee students, is defined as qualifying for free or reduced lunch. The income level for reduced-priced lunch is 1.85 times the poverty line, free lunch is 1.35 times the poverty line. Most low-income students qualify for free lunch.
Enrollment in the Choice Program. Enrollment statistics for the Choice Program are provided in Table 1. Enrollment in the Choice Program has increased steadily, but slowly, never reaching the maximum number allowed by the law. September enrollments have been 341, 521, 620, 742 and 830 from 1990-91 to 1994-95. The number of schools participating was: 7 in 1990-91, 6 in 1991-92, 11 in 1992-93, and 12 in the last two years. The number of applications has also increased, with again the largest increase in 1992-93. In the last two years, however, applications have leveled off at a little over 1,000 per year. Applications exceeded the number of available seats (as determined by the private schools) by 171, 143, 307, 238 and 64 from 1990-91 through 1994-95. Some of these students eventually fill seats of students who are accepted but do not actually enroll. The number of seats available exceed the number of students enrolled because of a mismatch between applicant grades and seats available by grade. It is difficult to determine how many more applications would be made if more schools participated and more seats were available. In 1992-93, when the number of participating schools increased from 6 to 11, applications rose by 45%. In the last two years, however, seats available increased by 22% and 21%, but applications only increased by 5% from 1992-93 to 1993-94 and declined this past year.
Learning About Choice and the Adequacy of Information on Choice. Table 2 indicates how survey respondents learned about the Choice Program. The results are fairly similar for the first four years. The most prevalent source of information on choice remains friends and relatives, which basically means word-of-mouth information. That informal communication is almost double the frequency of other sources. The first two yearly reports indicated this as a problem and suggested remedies including more funds for advertising the program. Additional monies were added by the legislature this year for that purpose.
Parental satisfaction with the amount and accuracy of that information, and with the assistance they received, is presented in Table 3. Satisfaction with the amount of information on the program in general is high in all years and this last year is almost identical to the first four years. There is, however, a difference in satisfaction of parents selected and not selected into the choice schools. Looking ahead to Tables 6a and 6b, when we create additive scales for these questions measuring satisfaction with administration of the choice program, there is a large
difference between those applicants who enroll in the choice program and those not selected.2 As indicated in the rows for Scale T3 (top panel, Table 6a), choice enrollees for 1990-93 have a mean dissatisfaction of 11.7, while non-selected parents have a much higher average dissatisfaction score of 14.9. As indicated in column 4 Table 6b, this is a highly significant result.
We do not have up-to-date data on the amount of information on the Choice Program that exists in the community at large. In our random sample survey in May 1991, we asked MPS non-choice parents if they had heard of the Choice Program. At that time, 51% responded that they had heard of the program.
Why Choice Parents Participated in the Choice Program. Table 4 provides the responses to survey questions rating the importance of various factors in parents' decisions to participate in the Choice Program. The results are consistent across the years. The leading reasons given for participating in the Choice Program was the perceived educational quality and the teaching approach and style in the private schools. That is followed by the disciplinary environment and the general atmosphere parents associate with those schools. Frustration with prior public schools, although not unimportant, was not as important a reason for applying to the Choice Program as the attributes of the private schools. At the bottom of the list are siblings in the school and the location of the school.
Demographic Characteristics of Choice Families and Students. The Choice Program was established, and the statute written explicitly to provide an opportunity for relatively poor families to attend private schools. In the first four years the program has clearly accomplished this statutory goal. Relevant demographic statistics are presented in Table 5, which, unless otherwise noted, are based on our surveys. For comparison purposes seven groups are identified including the most recent year applicants (1993), and then combined year samples for: choice applicants, choice students actually enrolled, students not selected, attrition students who could return to the private schools (i.e., they have not graduated) but do not, and the MPS control groups.
In terms of reported family income (Table 5a), the average income was $11,780 in the first four years. Applicants in 1993-94 were even poorer than the average (despite inflation). These numbers compare to a 1993-94 program income limit of $20,808 for the average family of 3, which is also the average in our MPS control group.3 Low-income MPS parents reported a somewhat higher family income, which averaged $12,100.
Consistent with the income statistics, in the first four years, 57% of the choice mothers reported being on AFDC or general assistance, compared with 39% of the MPS parents and 59% of the low-income parents.4 Employment rates for both full- and part-time employment were similar between choice and MPS parents. For the combined four years, 38% of choice mothers and 63% of the choice fathers were employed full time. These compared with 44% of the MPS mothers and 74% of the fathers. The low-income MPS figures were 33% for mothers and 59% for fathers.
In racial terms, the program has had the greatest impact on African-American students, who comprise 74% of those applying to choice schools and 72% of those enrolled in the first four years of the program (Table 5b). Hispanics account for 19% of the choice applicants and 20 % of those enrolled. The figures for the 1993-94 year are almost identical to the four-year averages. Both of these groups are disproportionately higher than the MPS sample. Compared with the low-income MPS sample, however, Hispanics are the most overrepresented, with Asians and White students the most underrepresented. The overrepresentation of Hispanics is due to the building and considerable expansion in capacity of Bruce-Guadalupe Bilingual School, which has a Spanish emphasis and is located in a neighborhood with an increasing Hispanic population. The school enrolled 43 choice students in 1990-91 compared with 166 in 1993-94.
The differences between those who applied and those who enrolled in the Choice Program (columns 1 and 3) for African Americans and Hispanics is statistically significant, but can also be explained by expansion patterns. Bruce Guadalupe has expanded most and also receives by far the most Hispanic applications. Therefore their acceptance rate is proportionately higher, while the mostly African-American schools on the North side turn away more of their predominantly African-American applicants.
In terms of marital status (Table 5c), choice families were much more likely to be headed by a non-married parent (75%) than the average MPS family (49%), and somewhat more likely than the low-income MPS parent (65%).5 The percentage has been almost identical for the four separate yearly cohorts.
One important difference to note between MPS and choice applicants is in family size (Table 5d). For the combined years, only 46% of the choice families report having more than two children. The average number of children in families applying to the choice program was 2.64. This compares with 54% of the MPS families having more than 2 children (2.95 children per family) and 65% of the low-income MPS families (3.24 on average). Again, the statistics on choice families are quite consistent in each of the four years. The slightly smaller family size of 1993-94 applicants may be due to the increasing number of available seats in the choice schools in the lower grades, thus increasing the probability that choice students will be first or second children.
A unique characteristic of the choice parents is that despite economic status, choice parents report higher education levels than either low-income or average MPS parents (Table 5e). Over half of the choice mothers reported some college education (52%), compared with 40% for the entire MPS sample and 30% of the low-income MPS respondents. The biggest difference in education appears in the category responding "some college." Although fathers more closely match the MPS control groups, they are also somewhat more educated. This finding was very consistent over three years, with only slightly lower education reported by the second-year choice families. In 1993-94, choice mothers are even more educated than for all the applicants in all years. On the other hand, choice fathers are not as educated as the four- year group. We have no explanation for these shifts, but the general pattern certainly is not changing.
The Importance of Education and Educational Expectations. Based on our measures, choice and MPS parents are similar in terms of the importance they place on education. We measured the importance of education relative to other important family values. The exact questions asked and percentage responses are contained in Appendix Table A1; scale descriptive statistics are in Table 6a; and the probability of scale scores being equal between groups are in Table 6b.6 The relevant scale is denoted as A1 to conform to Appendix Table A1. As confirmed in Table 6b, at standard levels of significance (p<.05) there were no significant differences between the importance of education for choice and the low-income MPS sample. Scores were even closer for choice and the total MPS sample. For some unexplained reason, there was a difference on this variable between enrolled and non-selected choice students. The latter report that education was less important than for families which were selected into the choice program.
Educational expectations are high for all groups, with choice parents somewhat higher than MPS and low-income MPS parents. Eight-six percent of choice parents in the first four years indicated that they expected their child to go to college or do post-graduate work. This compared with 76% of the MPS parents and 72% of the low-income MPS parents. Because sample sizes are large, these proportions are significantly different.
Experience of Choice Parents in Prior Public Schools. A more complete picture of choice parents includes the level of parental involvement, attitudes toward, and student success in their child's prior public school. Our surveys measured the degree of parental involvement in the school, the amount of parental help for children at home, and parent satisfaction with prior schools. The results are given in Tables 6a and 6b and Appendix Tables A2 to A6. The "grades" parents gave to their prior public schools are given in Table 7 and prior achievement test results are given in Table 8.
Based on four years of highly consistent data, the overwhelming conclusion is that choice parents are significantly more involved in the education of their children than MPS parents. Involvement was measured in terms of parents being contacted by schools, parents contacting schools, parental activity in school organizations and activities, and parents working with their children at home. For each measure in Tables 6a and 6b, the differences in the means of the scale scores (scales A2 to A5) between MPS low-income and choice parents were significant at greater than the .001 level.7 The greatest differences were for parents contacting their schools about their child's academic performance, classes, behavior, etc. (Table A2, Scale A2). There was less difference in parent organizational membership and activity (Table A4, Scale A4), but the difference was nevertheless statistically significant.
Choice parents also indicated significantly more involvement in their child's education at home through such activities as helping with homework, reading with children, etc. (Table A5; Scale A5). Another finding, however, was that their child's behavior was the most frequently cited reason for schools to contact parents (Table A2). This is consistent with parents expressing dissatisfaction with discipline in their prior schools (discussed below), and with discipline in private schools as one of the most important reasons given for why parents applied to choice schools.
Parental satisfaction with prior schools was also very different between MPS and choice parents. On every dimension, choice parents were less satisfied than the average or low-income MPS parent. The items in Table A6 go from most satisfied to least satisfied (note that a high score for scale A6 in Table 6a indicates dissatisfaction). What is distinctive is that the factors with which parents were most satisfied had little to do with the operation or outcomes of the school (textbooks, school location). On the other hand, the greatest dissatisfaction was with the amount the child learned and the discipline in the school. Further, in comparing the magnitude of differences between choice and MPS parents, the differences between the groups were greatest for those items on which choice parents were most dissatisfied. As with parental involvement, we can reject the hypothesis that satisfaction with prior schools is the same for MPS and choice parents with a probability exceeding .001.
Another indication of parental dissatisfaction with their prior MPS school was measured by a simple question that asked parents what grade they would give their prior school (on an A to F scale). The results are given in Table 7. As is apparent and consistent with other expressions of satisfaction, choice parents graded their prior schools considerably lower than MPS parents. That was especially true for the first-year choice parents, who on average rated their prior schools as a C. The second and third year averages for choice parents were 2.5 in both years, but the last year average grade for prior public schools rose to 2.7. In comparison, the MPS low-income and total control group rated their schools at a 2.8 average.
The attitudes of parents toward their children's prior public school within MPS may be a reflection of the fact that their children were not doing well in those schools. Prior test scores for choice applicants, the whole MPS control group, and the low-income MPS students, are given in Table 8.8 The test scores in Table 8 are Iowa Test of Basic Skills (ITBS) which are given in grades 1-8. The tests used are the Reading Comprehension Test and the Math Total Test. The latter consists of three subtests: Math Concepts, Math Problem Solving, and Math Computations.9 The results reported are for the last test taken while the student was in MPS. The reason for this is that we are trying to get as complete a picture as possible of a relatively small number of choice students. The majority of those tests are taken in the spring of the year of application (56% in 1990, 58% in 1991, 52% in 1992, and 54% in 1993).10
In all four years, ITBS scores taken in prior public schools by students applying to the Choice Program were significantly below the average MPS student taking the same test. The scores were also below the low-income MPS cohort in each year. In terms of differences of means for the Normal Curve Equivalents, the differences between choice and low-income MPS groups were significant at the .05 level for both tests in all years except the first year of the program. The differences were largest in the second year. The latest year prior scores for 1993-94 were lower than the previous year for both tests. Based on median scores, they were very low on math and similar to prior years on reading11
The absolute level of the scores indicates the difficulty these students were having prior to entering the program. The median national percentile for choice students ranges from 26 to 31, compared with the national median of 50. The Normal Curve Equivalent, which is standardized to a national mean of 50, ranges from 37.5 to 39.8, which is about two-thirds of a standard deviation below the national average. In short, the choice students in this program enter very near the bottom in terms of academic achievement.
Summary. Enrollment in the Choice Program has increased from 341 students in 1990 to 830 in 1994-95. The number of applicants exceeded the number of students enrolled in every year. The number of schools participating has increased from 7 in 1990 to 12 in 1994. The biggest limitation on the program is the number of seats available in the participating schools. Although the per-school expansion from 49% to 65% of the students enrolled being eligible for choice vouchers may increase numbers slightly, most schools are close to their capacity. Even if new schools participate, the program is unlikely to enroll the number of students permitted.
Parents learned of the program primarily from friends and relatives. In general, choice parents reported high satisfaction with the amount and accuracy of the information, and the assistance they received from the choice schools and the State Department of Public Instruction.
The overall portrait of choice students and families is complex, but very consistent over the years. The data clearly indicate that choice can be targeted toward poor families attempting to find an alternative to what they view as a poor educational environment for their children. The choice students come from poor, mostly single-parent households. Similar to MPS parents, approximately 60% are receiving AFDC or public assistance. The parents also express considerable dissatisfaction with prior public schools, and, based on prior test scores, there is clear evidence that their children were not doing well in those schools (both in relative and absolute terms).
Despite being poor, however, the choice families are smaller than those in the comparison groups, thus providing an opportunity for parents to focus more on any single child. In addition, the parents (especially mothers) are more educated, appear to have somewhat higher educational expectations for their children, and are more likely to work at home with their children on education problems. Finally, the choice parents participated in their children's prior schools at higher rates than the average parent.
This portrait presents something of a paradox. The Choice Program was specifically designed to provide an opportunity for poor parents to send their children to schools of their choosing which they could not otherwise afford. Four years of very consistent data indicate that in this it succeeds. In addition, there are numerous indications that these parents were dissatisfied with the public schools their children had been attending and that their children were not learning nearly as much as the average child in MPS. Thus one could argue these are exactly the types of families that should have access to an alternative source of education. One could also reasonably argue, however, that if these students and families remained in their prior schools, they could exercise considerable influence in attempting to improve those schools. Parents are educated, angry, involved, and have high expectations for their children. If engaged, and given the opportunity, they could push the public system rather than leaving it.
IV. THE CHOICE SCHOOLS
Very detailed case studies have been done on eleven of the twelve private schools in the Choice Program. In the First-Year Report, we described four choice schools in considerable detail. We also described one choice school that withdrew from the Choice Program and subsequently went bankrupt. In the Second-Year Report, because no new schools had joined the program, we briefly repeated the description from the first report and updated several statistical tables on staff characteristics, staff turnover, etc. In the Third-Year Report, we described the eleven schools that were in the program in 1992-93. No additional case studies were done in 1993-94 and funding shortages may prevent a final case study of the remaining school. Therefore, what follows is a brief update on students, staffing and personnel statistics for the last year. In an effort to provide a comprehensive report, Appendix B is a copy of the school descriptions provided in last year's report.
Students. Students in the Choice Program range in age from 4 to 19 years old. Most of the students are in four K-8 schools. Of the 620 students enrolled in September 1992, 531 (86%) were in these schools. In 1993, the number rose to 612 of 742, but the percentage declined to 82%. For 1994-95, 697 of 830 (84%) were in the largest schools. Forty-six students were in the two alternative high school programs in 1992-93 and 80 students enrolled in the three alternative programs in 1993-94. In 1994-95 there were two alternative programs with 69 students (8%). The remaining students were in one middle school program, four Montessori schools, and one Waldorf school.
The racial composition of choice students by school is mixed. Four of the schools in 1994-95 were almost all African American. Four others are predominantly African American (above 70%). One school is 93% Hispanic, and the remaining three schools are more evenly integrated. This pattern is partly the result of conscious specialization on the part of schools (for example, African-American cultural schools and a bilingual school); and partly the result of location. One well-integrated school has a formal policy of insuring that its student body matches its carefully defined community area in terms of race, ethnicity, gender and economic class. Several of the schools with relatively high tuition expressly entered the Choice Program to provide some cultural diversity in their student body.
Staffing. The staffing of the schools was examined in the First-Year Report and several issues were raised. One problem was turnover-of both teachers and administrators. Associated with that problem was the number of relatively new teachers in the schools. As indicated in Table 9a, staff turnover in 1991-92 continued to be a problem, and the number of new personnel increased. A third of the new personnel, however, were filling expansion positions. Both turnover and new personnel rates for 1992-93 were down substantially indicating some personnel stability in these five schools. In 1993-94, based on changes in eleven schools, the turnover rate remained constant at 18%, but the new personnel rate increased to 29% based on the creation of 10 new positions in addition to 16 replacements. For 1994-95, based on reports filed by nine schools, turnover increased to 24%, but the new personnel rate declined to 22% because there was one less classroom teacher in the schools. As reported in previous reports, administrator turnover has been quite high in the choice schools. Again in 1994-95, there are 4 new principals or administrators in the twelve schools.
Certification data are presented in Table 9b. In 1991, in the original five schools studied, 62% of the teachers had state certification. In the additional six schools, only 54% were state-certified. Of those 54%, 21% had both state and specialty certification. Forty-three percent had just specialty certification. The high percentage of specialty certification was due to the addition of four Montessori programs and a Waldorf school. These schools have national and/or internationally recognized certification programs for their relevant pedagogical specialties. For the nine schools reporting in 1994-95, 64% of the teachers were Wisconsin certified, with 20% having other certifications.
Seniority of teachers in the schools added since 1991 indicates more stable staffing. The average teacher seniority in these schools was 6.5 years, compared with 4.2 years in the original five schools studied in 1991. Twenty-six percent of the teachers had 10 or more years of experience in their current schools. Obviously, this still differs considerably from the seniority achieved in most public school systems. Again, as in 1991, pay and benefit levels were the most often cited reasons for considering leaving their current schools. Seniority data were not collected in 1994-95.
The racial composition of teachers has fluctuated as schools replace teachers and expand, and as new schools enter the program. As depicted in Table 9c, in the first year, the staff in these schools was predominantly white (75%) and female (89%). In the subsequent year, the teaching faculties became more diverse. In 1991-92, 71% were women and 38% were minority teachers (27% African-American, 11% Hispanic). In 1992-93, for the eleven schools, 82% were women and 30% were minorities. For 1993-94, based on all twelve schools, there was a slight increase in the number of men (up to 23%) and the minority percentage remained constant at 30. In 1994-95, the percentage of minority teachers in the nine reporting schools went up considerably - to 37%. The number of male teachers was similar to the previous year (24%).
V. OUTCOMES
We discuss five types of outcome measures in this section: (1) achievement test results; (2) attendance data; (3) parent attitudes; (4) parental involvement; 5) attrition from the program; and 6) a comparison of choice students over four years with those students who applied to choice but were not selected into the program. The legislation specified that suspension, expulsion, and dropping out also be monitored. Those measures, however, would be meaningful only at the high-school level.12 The high-school-level choice schools consist of alternative education programs for seriously at-risk students. Therefore it is unclear what the relevant MPS comparisons would be.13 We report only on the six areas indicated above.
Achievement Test Results
Cohort Test Results. Table 10 provides the aggregate test results for 1991 to 1994 for choice students and for students taking tests in MPS in the respective years. Tests were administered in April or May of each year. The results may be compared only crudely with those in Table 8, which indicated prior test scores for students accepted into the Choice Program. The prior test data in Table 8 were mostly from the previous spring, but were based on the last prior test in the student's file. As stated above, those data indicated the choice applicants were clearly behind the average MPS student, and also behind a large random sample of low-income MPS students.
The most relevant comparisons are with the low-income MPS subsample. Table 10 indicates that in reading the choice students tested in 1991 did better than the MPS low-income group, but in math they did somewhat worse. Tests taken in the choice schools in the second year-spring 1992- were considerably lower. They were lower than the scores in both the full MPS sample and among the low-income MPS students. Comparing the more relevant low-income MPS and choice students, the choice students mean math NCE was more than five points lower; the reading score, which was higher in 1991, was two points lower. Median national percentile ranks closely parallel these differences. For 1993, the scores of choice students were approximately the same in reading as in the prior year, but were considerably higher in mathematics.14 Although the reading scores were lower than the low-income MPS sample, the math scores have the same median National Percentile Ranking, but were 2.3 Normal Curve Equivalent points higher.
The 1994 Spring scores for all choice students were slightly higher than the 1993 choice cohort, and very similar to the MPS low-income group on both reading and mathematics. In terms of both median National Percentile Rankings and Average NCEs, the results between choice and low-income MPS students were almost identical. In contrast to the first two years, for both 1993 and 1994 choice students appear to be doing better in math than in reading (which is the pattern for MPS students in all years).
Because the cohort scores do not report on the same students from year to year, the most accurate measure of achievement gains and losses are change scores.
Change Scores. When we analyze those choice students for whom we have year-apart test scores for the four separate years, the results differ somewhat.15 By comparing how well students did compared to national samples, we can estimate how individual students changed relative to the national norms over the year. Descriptive change scores are depicted in Tables 11a to 11d for the respective years. They are also graphically portrayed in Figures 2a and 2b. We caution the reader, as we have in previous reports, that sample sizes in some years are quite small for choice students. The results in the table are based on differences in NCEs, subtracting the first-year score from the second.16
For the first year, the positive result was that the averages for all groups, except math scores for choice students improved. For the choice students, the gain was considerable, although not significant in reading. Math scores stayed essentially the same. The MPS numbers indicate considerable and statistically significant improvement in average math scores, with smaller gains in reading. Both low-income and non-low-income MPS students gained in math. Three of these gains were statistically significant due in part to larger sample sizes than for the choice students.
There were quite different effects in the second year (Table 11b). Change scores for choice students in math, and for MPS students in both reading and math, were not appreciable. None of these differences approached standard levels of statistical significance. In contrast to the first year, however, reading scores dropped for choice students. The decline was 3.9 NCE points for all students between 1991 and 1992. Because NCE scores are based on a national norm, this means that choice students scored considerably below where they were relative to the national sample in the prior year. The decline was statistically significant at the .001 level.
The results shifted again in the third year. Choice students declined slightly in reading, which was not significant. On the other hand, for the first time, math scores for choice students improved. The mean math NCE went from 38.3 to 42.7 for a 4.4 NCE gain which was statistically significant. Scores for MPS students, on the other hand, declined for both tests and for both groups. Because of relatively large sample sizes for the MPS control group, the decline in math scores was significant and estimated to be 1.2 NCEs for both the total MPS control group and the low-income sample.17
The results for 1994 again shifted for both groups. For all groups, reading scores effectively did not change. The same was true of math scores for the MPS groups, although the low-income MPS scores decline slightly more than the non-low income group. For the choice students, after a large math increase in 1993, there was a decline of 2 NCE points in 1994. Because of attrition, graduation, and new entries into the Choice Program, the characteristics of the student body differ in the Choice Program each year. When the impact of factors which in part control for those changes are taken into account, there are no significant differences in the achievement gains between choice and MPS students in 1994. Regression analysis allows us to control for those additional factors.
Regression Analysis. Because test score differences could be based on a number of factors, and the factors could be distributed differentially between choice and MPS students, to provide an accurate and confirming picture of test achievement, it is necessary to control statistically for factors other than if students were in MPS or choice schools. Those controls are provided by multivariate regression analysis of the combined MPS and choice samples. There are a number of ways to model achievement gains. The most straightforward is to estimate the second-year test score, controlling for prior achievement and background characteristics and then including a (dichotomous) variable to measure the effect of being in a choice or MPS school. Tables 12 to 15 give the results for 1991 to 1994. Tables 16a and 16b contain more detailed and comprehensive models which include multiple prior tests and the effects of being in the Choice Program for differing lengths of time.
In general the models fit quite well with the raw change scores depicted in Tables 11a to 11d. The b columns in Tables 12 to 15 contain the coefficients that determine the effects of the variables on predicting the dependent variable (the relevant test). In both tables, a critical coefficient will be the "Choice" (dichotomous) variable in the second row from the bottom of the table. Dichotomous variables have a value of 1 if the student has the characteristic, and 0 if they do not. The effects of the coefficient can be read as a straight difference in the NCE score on the respective test being estimated. Thus, for example, in Table 12, being a low-income student has an estimated negative effect of 4.14 NCE points on the 1991 reading score.18 Similarly, the effect of being in a private school in the Choice Program (as compared to being in MPS) produced an estimated increase in Reading for 1991 of 2.32 Normal Curve Equivalent points. These predicted effects occurred after simultaneously controlling, or in essence keeping constant, all other variables in the model.
The only variables in the models that are not dichotomous variables are the prior-year test and the grade level of the student. The effects of these variables are also easy to interpret. They can be read as the effect that a one point difference in the variable has on the predicted test. Thus, for reading in Table 12, if a student is one NCE point better on the prior (1990) reading test than another student, we would predict he or she would do .59 points better on the 1991 test. For "Test Grade" for each additional grade students are in, we would estimate a lower 1991 reading score of .61 NCE points -- lower because the sign of the coefficient is negative.
The probabilities indicated in the tables are a statistical measure of how likely the coefficients are to differ from 0. Traditionally, those coefficients that have a probability of .05 or less are considered "significant." That is an arbitrary convention, however, and we therefore provide exact probabilities.
A number of patterns are very consistent across the four years and two tests. First, prior tests always are highly significant predictors of current year tests. And the range of the prior test coefficients are not only consistent, varying from .59 to .66, but also consistent with prior test results in other studies estimating prior achievement effects.19
Second, test grade and low-income have consistent effects although there is variation in the size of the coefficients and some variation in the probabilities of the estimates. In both cases, the effects are negative. Specifically, if a student comes from a low-income household, the effect ranges from lower math score in 1992 of 1.93 NCE points (Table 13) to a lower math score in 1994 of 5.03 NCE points (Table 15).
Gender and race seem to have mixed effects although the direction of the effects is generally consistent. Girls tend to do better than boys (positive sign for the b) on all tests but the 1991 Math test. And the difference is considerably larger, and more likely to be significant, in reading than math. For race, being African American or Hispanic tends to predict lower scores on most of the tests, although again the size of the coefficients and their accuracy vary from year to year. In the most recent year (1994, Table 15) the race effects are lower than in prior years and essentially zero for Hispanics.
The effects of being in the Choice Program can be summarized for each test and year by noting the estimated effects of being in the Choice Program as indicated by the b coefficient for "Choice" in Tables 12 to 15.
Effects of Being in Choice Rather than MPS, 1991 - 1994
Estimated
NCE Probability
Year Test Difference b = 0
1991 Reading +2.32 .106
1991 Math -2.07 .166
1992 Reading -3.35 .001
1992 Math -1.23 .255
1993 Reading + .76 .386
1993 Math +3.86 .000
1994 Reading - .30 .752
1994 Math - .95 .376
The results, consistent with the raw change score differences between the groups as reported in Table 11, suggest a mixed and changing pattern of achievement results. By conventional standards of statistical significance (p<.05), only the negative reading score in 1992 and the positive math score in 1993 are significant. Stretching that a bit, we conclude that the first year score differences are appreciable but go in opposite directions. For 1994 there clearly were no appreciable differences in achievement between choice and MPS students.
A more complex and comprehensive analysis is presented in Tables 16a and 16b. These tables present just the b coefficients and the standard errors which together determine if the estimated effects are statistically reliable. We present models for each year, with separate indicators for the number of years students were in the Choice Program. These models include both reading and math prior test scores for each test being predicted. This provides a more complete measurement of prior achievement. As is apparent, both prior tests are good predictors of the post-test results, but as expected, the relevant test has a much higher coefficient (i.e., math prior for math; reading prior for reading). Again, these parameter estimates are very close to the effects of prior achievement tests in other studies.20
Because prior achievement is being measured more comprehensively, the effects of other variables are attenuated -- the coefficients are smaller -- relative to the results in Tables 12 to 15. However, the patterns of significant results do not change. For reading, girls clearly do better; but not for math. African Americans do worse than other groups, but the effects are less in the fourth year. Hispanics do worse in the first year of the program, but no worse than whites in most later years. Being poor has a consistently negative effect, but it varies in magnitude from year to year.
The key test in these tables is the longitudinal effect of being in the Choice Program. The dummy "Choice" variables indicate students who have been in the program 1,2,3, or 4 years. The results, unfortunately, do not provide a consistent pattern. In the first year in the program, obviously everyone was a first-year choice student. In the second year (1992), second-year students did worse than first-year students on reading and the difference was significant. In 1993, the now third-year students again did worse on reading, but the second-year students had the highest scores. Both the second and third-year students did better on mathematics, with statistically significant advances relative to MPS students. The fourth year is inconclusive because of small changes in test scores for all groups, but it appears that the advanced choice students did better than the first-year students.
Thus there is some mixed and relatively weak evidence that students who remain in these schools do somewhat better than students who do not. When we analyze students who continue and those who leave the program, however, the pattern is relatively clear, and indicates that poorer students tend to leave the program while better students continue.
Summary. Scores for both choice and MPS students have fluctuated from year to year. For 1994, there were not significant changes in scores for either choice or MPS students. Although choice student scores on math declined after a previous year of significant improvement, when we controlled for other relevant variables, the decline was insignificant. Thus, as with last year's conclusion, it appears that choice and MPS students do not differ in any predictable way on achievement tests over the first four years of the Choice Program.
Estimation of the effects of being in the Choice Program for multiple years was also not conclusive. The results were negative in the second year, but appeared to be more positive in the last two years. Few of these results, however, were statistically significant and they did not take into account attrition from the program.
Attendance
Attendance is not a very discriminating measure of educational performance at this level because there is little school-to-school variation. For example, in the last three years, average attendance in MPS elementary schools has been 92% in each year. Middle school attendance for the same years averaged 89, 88 and 89%. Attendance of choice students in the private schools (excluding alternative schools) averaged 94% in 1990-91 and 92% in 1991-92, which puts them slightly above MPS, but the differences were obviously slight. In 1992-93, excluding SER Jobs and Learning Enterprises, attendance at the other schools was 92.5%. For 1993-94, attendance in the nonalternative schools (thus excluding Exito and Learning Enterprises) was 93%. It can be concluded that overall attendance was satisfactory and on average not a problem in choice schools.
Parental Involvement21
Parental involvement is stressed in most of the choice schools and, in fact, is required in the contracts signed by parents in several of the schools. Involvement takes several forms: 1) organized activities that range from working on committees and boards to helping with teas and fund raising events; 2) involvement in educational activities such as chaperoning field trips, and helping out in the classroom or with special events.
Tables 6a and 6b provide four-year data on parental involvement scales for choice parents both in their prior public schools and in the private schools their students attend under the Choice Program. Table 6a provides the means and standard deviations of the scales for the applicants from 1990 to 1993 and the comparable scale scores for choice parents from 1991-1994 June surveys. Table 6b provides statistics on differences of means between relevant groups. Frequency distributions and the exact questions comprising each scale are presented in Appendix Tables A2 to A5.
For the four types of educational involvement we measured, already very high levels of parental involvement significantly increased in the private choice schools (Table 6a, Scales A2, A4, and A5). These include parents contacting schools, schools contacting parents, and parental involvement in organized school activities. The increases were significant at the .001 level (Table 6b, column 4). The one exception was parent's involvement in educational activities at home (Scale A5). For that scale involvement increased but the increase was not statistically significant (Table 6b, Scale A5, column 4). These results have been confirmed independently in each of the four years of the program.
Parental Attitudes
Parental Satisfaction With the Choice Schools. In all four years, parental satisfaction with choice schools increased significantly over satisfaction with prior public schools. School satisfaction is indicated by scale A6 in Tables 6a and 6b. As described above, satisfaction with their prior schools was significantly less than satisfaction of the average or low-income MPS parent. Reported satisfaction with the choice schools is indicated in these tables as "choice Private School 1991-94" and was measured on the spring surveys in each year.
Parental satisfaction with the private choice schools dramatically exceeded satisfaction with prior MPS schools. As indicated in Table 6b, the scale difference is highly significant. As indicated in Appendix Table A6, based on the percentage of parents responding as "very satisfied," attitudes were more positive on every item. Comparing the panels for "Choice Enrolled, 1990-93" (which measures satisfaction in prior MPS schools) to "Choice Private Schools, 1991-94" in Table A6, gain on all the measures but one was between 15 and 17% in those "very satisfied." The only item in which gains were modest (1%) was for "teacher performance." Thus the parents who responded to our surveys believe they found in the choice schools what they professed they were looking for when they entered the program-increased learning and discipline.
Another indication of parent satisfaction is the grade parents give for their children's school. On the follow-up survey in June of each year, we asked parents to grade the private school their children attended. The grades, which indicate substantial differences with the grades they gave their prior MPS schools, are given in Table 17. For the four-year period, the average prior grade (on a scale where an A is 4.0) improved from 2.4 for prior MPS schools to 2.9 for current private schools. The overall grades have been relatively consistent for each year and always above the grades given to prior MPS schools.
Attitudes Toward the Choice Program. Follow-up surveys in June of each year asked parents of choice students if they wanted the Choice Program to continue. Respondents almost unanimously agreed the program should continue. Over the four years, 98% of the respondents felt the program should continue, with 98.7% expressing that belief in 1994.22
Attrition From the Program
Attrition Rates. One of the issues concerning the Choice Program is the rate of attrition from the program. Attrition rates, calculated with and without alternative high school programs are presented in the last two rows of Table 1. Attrition from the program is not inconsequential, although there appears to be a clear downward trend. Overall attrition, defined as the percentage of students who did not graduate and who could have returned to the choice schools, has been 46%, 35%, 31%, and 27% in the first four years. Excluding students in alternative programs, the rates are 44%, 32%, 28%, and 23%. As the program has continued and changes have taken place in the schools - including one brand new facility (see Appendix B), the attrition rate has declined. The key questions are: How substantial is the rate? Who is leaving? For what reasons? And where are they going?
How High are the Rates? In one sense, the answer to the question of how high the attrition rates are depends on the relative movement of students in and out of comparable Milwaukee public schools. Unfortunately, Milwaukee only reports data on transfers between schools from September until June. The average "mobility rate" for this period for comparable schools in MPS is approximately 33 percent. That figure, however, includes both "in" and "out" transfers from schools between September and June. If we assume there were equal numbers of in and out transfers in MPS, the comparable rate would be 16.5 percent.
The Choice Program includes only "in" transfers for students who are wait listed, and subsequently fill seats of accepted students who do not show up in September. Thus excluding the first year when Juanita Virgil went bankrupt, attrition during the year was less than MPS in all subsequent years based on students leaving from September to June. Unfortunately, MPS does not compute statistics on school transfers over the summer, although data are available in their Student Record Data Base. Using these data, we have tried to refine a study of mobility throughout the year in MPS. What we would like to do is determine the percent of "unnatural" moves that occur in any given year. Natural moves would include moves between schools when a student completes the highest grade in the school. As reported above, data are reliable in predicting moves within the year, all of which are assumed to be "unnatural." Because of the elaborate choice mechanisms in MPS and because of data entry and record keeping problems at the beginning of the school year, it is extremely difficult to compute any more than an estimated range of mobility for the whole year.23 Combining within-year and summer moves, we estimate that between 22% and 28% of MPS kindergarten through eighth grade students change schools each year.
Assuming this range is accurate, and our within-year numbers closely match the within-year elementary school mobility MPS reports, what they seem to indicate is the following: 1) mobility out of the choice schools may be very close to the norm for MPS; and, 2) for both systems, the data suggest considerable problems. If education is a long-term process that is most successful in a continuous school environment, this level of mobility will make that process difficult.
Who is not returning to choice schools? After three years, we have information on enough students to provide a portrait of the differences between students who continue in choice and those who leave. Differences in the two groups are presented in Table 18. The table includes demographic, school distance, achievement test, and attitudinal and parental involvement data.
Demographically, girls were more likely to remain in the program (56% of continuing students were girls; 52% of those who left were girls). African Americans (75% compare to 71%) and Whites (7% compared to 4%) were more likely to leave than Hispanics (22% stayed, 16% left). Family income and other measures of socioeconomic status were tested and there were no substantial differences between the groups.
As is reconfirmed in survey data provided below, those who leave the Choice Program traveled farther to their choice schools than those who did not. Those who left traveled an average of 3.36 miles to their private school, compared to the 3.07 miles for those who remained. Achievement data also indicate clear differences between those who left and those who continued in the Choice Program. Those who left had lower test scores and lower test change scores on both reading and math. Both prior reading and prior math scores were lower than those who continued. The ranges of significance are from .11 to .07 on two-tailed tests, but the direction is clear. Tests taken in the spring in the choice schools (post) were also lower for those who left. The most telling finding, however, may be that the change scores of those who left were considerably lower than those who remained. Thus the students who left the private schools started behind, they remained behind, and they apparently learned less than the students who continued.
Attitudinal and parental involvement measures differed on only one important dimension - satisfaction with the choice school. Based on spring surveys of attitudes toward and involvement in the choice private schools, there was no difference between the two groups on involvement measures or a measure of the importance of education. Those who left, however, indicated considerably greater dissatisfaction with the school than those who continued. The dissatisfaction scale means differ at a .001 level of probability.
The portraits of those who left and those who stayed over three years make sense. Hispanics, who have a new school in Bruce Guadalupe, tend to stay with the program somewhat more than whites or blacks. Those who live closer are more likely to stay in the schools. And those students who begin with relatively high achievement and continue doing better are more likely to remain. The leaving parents are clearly less satisfied with the schools than those who remain. This portrait is confirmed by parent's responses to why their children left the program.
Why are students not returning to the choice schools? Because analysis of the causes of attrition was not part of the original study design, by the time we realized how many students were not returning after the first year, we were unable to follow up with non-returning families. That was rectified in the following years by using very brief mail surveys or telephone interviews. The surveys and interviews simply asked where the student was currently enrolled in school (if they were), and (open ended) why they left the choice school. The response rate to our inquiries has been 38%. This rate of return is slightly lower than to our other surveys. The normal problems of mail surveys are compounded by the fact that we do not know who will not return until enrollment actually occurs in September. Thus in many cases addresses and phones numbers are not accurate. The largest bias in our responses is undoubtedly families who moved out of the Milwaukee area and did not leave forwarding addresses. Telephone searches were impossible for that group. Our results should thus be treated with some caution.
The reasons parents gave for leaving are presented in Table 19. Beginning at the bottom of the table, approximately 15% of the responses (and they could give more than one) indicated child or family specific reasons--including moving. We suspect that this category is underestimated since we undoubtedly were not able to locate as high a proportion of families who moved out of the area.
Almost all of the remaining responses were critical of some aspect of the Choice Program or the private schools. The leading problems with the program were the lack of religious training, which is not allowed by the statute, references to transportation problems, and difficulties in reapplying to the program. Within-school problems most often cited were unhappiness with the staff, usually teachers, dissatisfaction with the general quality of education, and perceptions that discipline was too strict. The lack of special programs, which might have been available elsewhere, was also cited in 6% of the responses.
Thus, as we would expect, survey responses fit with the factors that seem to distinguish attrition students from those who remain - distance and transportation problems, less achievement success, and resulting dissatisfaction with the private schools.
Where are the attrition students? Our data indicate that approximately half of the students who left after the second and third year (57%) enrolled in MPS schools, 26% in other private schools in the area (often for religious reasons), with the remaining 16% going to MPS contract schools, home schooling, or schools outside Milwaukee.
Summary. In summary, attrition from the Choice Program is declining and is similar to the rates of movement between schools for MPS students, but is undoubtedly a problem for both sets of schools. It must be remembered, however, the large majority of choice students in the most recent years do not leave the schools each year - and that percentage seems to be growing. The picture of those who left is understandable. That is, leaving was a predictable response by parents who were dissatisfied with their students' achievement progress and other aspects of the school.
Nonselected Choice Applicants
Students not selected into the Choice Program in the random selection process represent a unique research opportunity. Because parents of these students have exercised their option to attempt to enroll their children in the program, if there are any unmeasured characteristics of families seeking private education, they should, on average be similar between those in and not in the program. This unmeasured selectivity bias has always been a problem in ascertaining differences between public and private school achievement.24 Although we have sporadically reported characteristics of these groups in previous reports, because sample sizes have been so small, we have not systematically reported on the differences between these students and the choice students.
We survey not selected choice parents as part of the applicant survey in the fall of each year. We also attempt to resurvey them in the spring. Sample sizes are small due to the small number of nonselected students and low survey response rates. As is apparent in Table 6a, nonselected student samples over four years are approximately 200 for either fall or spring surveys. We also have prior test scores taken in MPS and test scores taken in MPS if the nonselected students returned to MPS.25 We compare choice and not selected choice students and families on most of the same variables we used to analyze choice students relative to our MPS control group.
Demographic Differences Between Selected and Nonselected Choice Students and Families. As indicated in Table 5, race is the only significant demographic difference between those selected and not selected into the Choice Program. As explained earlier in the report, this difference, which favors higher enrollment of Hispanics, is based on the increasing size of the major Hispanic school in the program. Hispanic students applied to that school and because the space was available, they were not turned away. The only other relevant difference in the groups is the distance they traveled to their schools. This is presented in Table 20. In terms of miles, choice students attending private schools lived approximately 3.16 miles from their schools. Nonselected choice students who returned to MPS schools lived 3.98 miles from their schools.
Achievement Test Differences. As indicated in Table 20, there were no significant differences in achievement test results between choice students and those who were not selected into the choice program. Both prior and post-math scores were almost identical for the two groups. Prior reading scores, undoubtedly reflecting the higher percentage of Hispanics in choice, were higher for the not-selected group (although not significant at conventional levels). The reading change scores, however, were very close for the two groups (-.88 for choice and -.44 for nonselected choice students over the four years).
Parental Involvement. In terms of prior involvement in public schools, nonselected choice parents looked very much like selected choice parents. As indicated in Column 3 of Table 6a, none of the parental involvement scale means differed from choice parents in a statistically reliable way. Both nonselected and selected choice parents had considerably higher levels of prior parental involvement than MPS parents. However, as indicated in Table 20, with one exception, the parental involvement of nonselected parents was lower in their subsequent schools than the involvement of choice parents in the choice private schools. The one exception was involvement with their child at home. The higher school parental involvement of choice parents further reinforces the conclusion that the choice schools emphasize involvement, including requirements to participate through contracts, fund raising, etc.
Parental Attitudes. Finally, there are significant differences between choice and not selected choice parents on the importance they place on education and, especially on the satisfaction they express with their schools. Choice parents began with higher scores on our importance of education scale (Table 6b, Col. 3), and when measured on spring surveys over the four years, that difference remained. Dissatisfaction with prior schools was slightly greater for not selected choice parents when they applied to choice (Table 6a, Scale A6), however, the differences were not statistically significant (Table 6b, Col. 3). As indicated in Table 20, both sets of parents were less dissatisfied with subsequent schools, but the choice parents satisfaction improved much more than the nonselected choice group. These differences in dissatisfaction were highly significant.
Summary. In terms of demographic characteristics, nonselected choice students came from very similar homes as choice parents . They were also similar in terms of prior achievement scores and parental involvement. In terms of achievement outcomes, test score changes were very similar between the two groups -- never approaching statistically significant differences. Thus there was no difference in terms of achievement between those who got into the program and those who did not. This parallels the test score findings over four years comparing choice students with our control group.
However, on parental involvement, the importance parents place on education, and satisfaction with private or other subsequent schools, choice parents appear to be significantly higher. These measures all come from our second wave spring surveys with the attendant cautions noted. However, again the findings exactly parallel the differences noted in our random sample control group. Choice parents were more satisfied with their schools, more involved and place slightly more importance on education than on other goals.
Outcome Summary
Outcomes after four years of the Choice Program remain mixed. Achievement change scores have varied considerably in the first four years of the program. Choice students reading scores increased the first year; fell substantially in the second year, and have remained approximately the same in the third and fourth years (See Figure 2a). Because sample size was very small in the first year, the gain in reading was not statistically significant, but the decline in year two was. In math, choice students were essentially the same in the first two years, but recorded a significant increase in the third year, but that was followed by a significant decline this last year.
MPS students as a whole gained in reading in the first two years, with a relatively small gain in the first year being statistically significant. There were small and not significant declines in the last two years. Low-income MPS students followed approximately the same pattern, with none of the changes approaching significance. Math scores for MPS students were extremely varied. In the first year there were significant gains for both the total MPS group and the low-income subgroup. In the second year, the scores were essentially flat, but in the third year they declined significantly. Again, in the fourth year there was essentially no change in either the total MPS or low-income MPS groups.
Parental attitudes toward choice schools, opinions of the Choice Program, and parental involvement were very positive over the first four years. Attitudes toward choice schools and the education of their children were much more positive than their evaluations of their prior public schools. This shift occurred in every category (teachers, principals, instruction, discipline, etc.) for each of the four years. Similarly, parental involvement, which was more frequent than for the average MPS parent in prior schools, was even greater for most activities in the choice schools. In all years, parents expressed approval of the program and overwhelmingly believed the program should continue.
Attrition (not counting students in alternative choice schools) has been 44%, 32%, 28%, and 23% in the four years of the program. Estimates of attrition in MPS are uncertain, but in the last two years, attrition from the Choice Program was comparable to the range of mobility between schools in MPS. Students who left were very similar to those who continued in terms of parental involvement and family demographics (income, family status). Those who left the program did have lower prior test scores, lower scores in the private schools, and lower change scores than students who returned. They also lived farther away than continuing students. Finally, the parents of attrition students expressed lower levels of satisfaction with the schools. The reasons given for leaving include complaints about the Choice Program, especially the limitation on religious instruction and problems with transportation. They also include complaints about staff, general educational quality and the lack of specialized programs in the private schools. We probably underestimate the number of students who left for family-specific purposes, such as moving out of the area, because we were unable to locate these families. Based on follow-up surveys and interviews, we know that approximately half of the students appear to be returning to MPS schools, with most of the rest going to other private schools.
We also evaluated the differences over four years between choice families and students and students who applied to the program but were not selected. In terms of outcomes, test score changes were very similar between the two groups -- never approaching statistically significant differences. Thus there was no difference in terms of achievement between those who got into the program and those who did not. However, on parental involvement, the importance parents place on education, and satisfaction with private or other subsequent schools, choice parents appeared to be significantly higher. Choice parents were more satisfied with their schools, more involved and place slightly more importance on education than on other goals. These findings exactly parallel the differences noted between choice and our MPS random sample control group.
VI. CONCLUSIONS
We have tried to provide a very broad array of data on the Milwaukee Parental Choice Program so that those who read this report can form their own opinions based on the criteria they deem most important. For us the main issues and questions have been: (1) Does the program provide alternative educational opportunities for relatively poor families? (2) In creating these opportunities has the program harmed the existing system of public education in Milwaukee? (3) Have the outcomes of the program been successful in terms of providing educational benefits for students, parental satisfaction, and benefits for the choice schools?
Does the Program Provide Alternative Educational Opportunities? As we have noted in all of the reports on this program, the answer to the question is yes. Private education is probably beyond the means of the vast majority of the families applying to the Choice Program. The families are unhappy with their prior experiences in public schools, and their children are not doing as well as most children in the public schools.
Does the Program Harm the Existing System of Public Education in Milwaukee? The program enrolled 830 students in 12 schools on the third Friday in September 1994; the Milwaukee Public School system was teaching over 96,000 students on the same day. The students in the Choice Program were not the best, or even the average students from the Milwaukee system. Thus both in terms of size and the potential for "creaming," this program, as currently configured, poses little threat to the MPS system. Choice parents, however, were more educated and more involved with their prior public schools than average MPS parents. Thus the loss of these (few) parents can be construed as potentially detrimental to engaging and involving parents in the public schools.
Have the Outcomes of the Program Been Successful? The answer to this question remains mixed. In terms of achievement test scores, the answer is that students perform approximately the same as MPS students. Those who leave the program come in with lower test scores, and leave still behind. On the other hand, attendance of choice children is slightly higher, and overall parental satisfaction with the private schools and the program is high. For the schools, the program has generally been positive. It has allowed several to survive, several to expand, and contributed to the building of a new school which opened in 1993.
Honorable people can disagree on the importance of each of these factors. One way to think about the program is to ask whether the majority of the students and families involved are better off because of this program. The answer of the parents involved, at least those who respond to our surveys, was clearly yes. This was despite the fact that achievement, as measured by standardized tests, was no different than the achievement of MPS students. Obviously the attrition rate and the factors affecting attrition indicate that not all students will succeed in these schools, but the majority remain and applaud the program.
What Does This Study Not Address? This program provides very limited evidence for evaluating or anticipating the effects of more inclusive choice programs. Programs proposed in other states, such as the tuition voucher initiatives in Colorado or California, did not have the same conditions for eligibility or school selection procedures as this program. Furthermore, families taking advantage of choice opportunities in these relatively unconstrained programs could well be very different from those in the targeted Milwaukee program.
Expansion of this program to include religious schools, which is probably the only feasible way to enlarge the program significantly, pushes the program into uncharted waters and involves a whole new set of issues. These would include constitutional issues at both the state and federal level, questions of governmental subsidy and regulation of religious organizations, fiscal issues, and political questions concerning how far and with what conditions the program would be expanded in the future. These are all very serious issues, none of which have been addressed in these studies.
VII. RECOMMENDATIONS
The majority of evidence in this report is consistent with and confirms the evidence presented in the first three year reports. The recommendations of those reports, although many were included in both the Governor's budget and the budget of the Department of Public Instruction in 1993, with several exceptions, were not acted upon by the legislature (See Appendix C for a list of those recommendations). We feel that most recommendations remain valid. Money was put in the budget for disseminating information on the Choice Program (as recommended in the first two years), but the amount was quite small and an analysis should be done to determine if more is needed. The program was also expanded in terms of the total number of schools and percentages of students in the schools who could participate in the choice program. Because the overall limitation of students is almost twice current enrollment and because the number of applicants has not grown appreciably, those changes seem adequate.
There are two changes in recommendations we believe should be considered, and one earlier recommendation we feel needs to be reinforced. First, we recommended in the first year that the schools be required to meet all statewide outcome assessment standards required of public schools. Since that time, the outcome and performance standards required of public schools in terms of statewide tests and school report cards have increased considerably. Although we applaud these changes based on the vastly improved quality of information they provide to parents, citizens, and government officials, some of the data required of public schools may be very difficult and costly for the private schools to generate. Thus we maintain the thrust of the recommendation -- that more information should be available on the performance of choice schools -- but do not advocate automatic adoption of all standards as they are currently promulgated.
Second, a number of the schools over the years have been lax in providing information in a timely manner to the Department of Public Instruction. Often this is simply the result of overworked and changing staffs in the private schools, but it does affect the timely and accurate administration of the program, and there is almost nothing DPI can do about it because the statute gave them almost no enforcement powers. That matter should be addressed in any revisions of the statutes and administrative rules governing the program.
Finally, perhaps the most pressing problem, which we have alluded to in every report, are needed changes in the transportation reimbursement system. We encourage the legislature to reconsider our prior recommendations on that and other issues.
1. Although the response rates (given in Appendix D) on some of our surveys are low relative to face-to-face interviews with national samples, they are higher than the approximately 20 percent response rates that MPS reports for its usual surveys. We have independent measures of race and qualification for free lunch from the Milwaukee Student Record Data Base for both the random sample and the choice students. Thus we were able to assess sampling bias and construct weights to offset that bias. For MPS the only statistically significant sampling bias was for race, where we had a less than expected response for African Americans, oversampling of Asians and Whites, and a slight undersample of low-income families. Appendix E provides race and income data on survey respondents. For choice students, there was a disproportionately high response from African Americans and a low response from Hispanics. The results presented in this report are for unweighted samples.
We have also analyzed scales and demographic variables using three weights: a weight based on expected race; a weight based on expected low/non-low income; and a weight combining both race and income. The combined race and income weight is the most accurate. Because for the MPS respondents, the sampling bias for race was considerably larger than income, the income-weighted analysis produced no significant differences except on the income variable itself. The race/income analysis produced only one marginally significant difference on attitude scale means. It also produced significantly different effects for household income, percent of female parents on AFDC, and for the percentage of single-parent families. These
differences are noted in footnotes in the text. Analysis of the weighted choice sample produced
no differences that approached significance at the .05 level.
2. Table 6a and 6b contain information on "scales" which are sets of questions measuring an underlying concept. We created simple scale scores by adding together responses to each item. The items are presented in Appendix Tables A1 to A6 along with the percentage responses. Table 6a contains statistics on the scales, define the scale direction, and report the Chronbach Alpha statistic for the scale. Chronbach Alpha is a measure of how well the items form a scale. It has a maximum value of 1.0 when all items are perfectly correlated with each other. The r column in Appendix Tables A1 to A6 indicates how well each item fits with the scale by measuring the correlation of the item with the scale score minus the item in question.
3. When we analyzed these data using a race/income sampling weight, the MPS average income dropped to $19,900.
4. The race/income weighted sample produced an MPS estimate of 42.5% of female respondents on AFDC or general assistance in contrasted to the 39% actually reported.
5. The race/income weighted sample estimate of single parent families was 55% which is significantly higher that the non-weighted MPS sample. By either measure, however, the difference in proportions are statistically significant between MPS and the choice families.
6. See footnote 2 for an explanation of the scaling methods we employed.
7. Differences with the total group of MPS parents on all scales were also significant at the .001 level. The only scale affected by using weighted observations was for Scale A5, Participation in Educational Activities. When the sample was weighted to offset race and income bias, the mean increased to 7.12, which was significant at the .05 level. The remaining scale means were almost identical to the unweighted means.
8. Test scores are not available for all students in either group because tests are not given every year in MPS. Therefore, there will be no tests for 4- and 5-year old kindergarten, and few for first grade students. Lateral entry at higher grades might also miss some students because primary testing is in grades 2, 5, 7 and 10. For the few high school students in the Choice Program, the 10th grade test was excluded because very few of these students were tested and because students were entering alternative schools (schools for students contemplating dropping out of school or pregnant teenage students).
9. A number of the tests taken in MPS are dictated by rules for the federal Chapter 1 program which requires testing in every grade in reading and math using a standardized achievement test. In 1993, the federal regulations changed from requiring total math, consisting of three subtests, to just "problem solving." With that change, MPS dropped Chapter 1 testing using all three subtests for some students. Fortunately, the correlation between the Problem Solving Component and the Total Math score is .88. We were able to use an estimated regression model with Problem Solving estimating Total Math for students taking just the Problem Solving portion. The details of this procedure are described in Technical Appendix F.
10. Because sample sizes are relatively small for choice students, the most reliable statistic in these tables is the mean Normal Curve Equivalent (NCE). Median and percent at or above 50% National Percentile Rankings are included because these statistics are routinely reported by MPS. Because a number of students may be bunched together with small samples, both of these numbers are volatile.
11. Difference of means tests indicate that all choice student scores were significantly lower (at the .001 level of significance) than those in the total MPS sample.
12. There is almost no dropping out at the elementary level. Drop-out rates are also extremely low in middle schools. In MPS suspensions are also rare in these grades and the policies and reporting vary considerably from school to school. For example, student fighting, which leads to a suspension for up to three days in most of the private schools, may end up with a student being sent home in MPS. Whether that becomes an official suspension or not may depend on the principal and the reactions of the child or parents. The numbers of official expulsions are even smaller than dropouts or suspensions. See John F. Witte, "Metropolitan Milwaukee Dropout Report," Report of the Commission on the Quality and Equity of the Milwaukee Metropolitan Public Schools, 1985.
13. The noncompletion rate for choice students in Ser Jobs for Progress was 52% (14 of 27) for the 1990-91; 62% (23 of 38) in 1991-92; and 72% (21 of 29) in 1992-93. For Learning Enterprises, the completion rate was 29% (5 of 17) for 1992-93. For 1993-94, 3 Learning Enterprises students graduated, 24 completed the year, 19 dropped out and 2 were expelled. All of the 19 ninth and tenth grade students in Exito School completed the 1993-94 year with an average attendance of 80%. What is difficult to determine, given that these are students seeking alternative educational environments, is what these numbers mean in terms of prospective success.
14. The 1993 choice scores are a correction from the score reported in last year's report. One school failed to test several classrooms in time to have their tests scored electronically by the Iowa Testing company. They therefore hand scored their tests. In so doing, there were several errors in converting grade equivalents to national percentile rankings. The errors resulted in lower scores for their students. We caught the error in preparing this year's report and have corrected the numbers. For several choice students who did not take all three math tests, we added estimated math total scores as has been done for some MPS students who just took the problem solving portion of the math test. Those additions did not change the aggregate results, but increased the N by 7 students.
15. Please note that the cohort population described in the last section is not identical with students for whom we have change data from one year to the next. Thus, tables 10 and 11 are not
directly comparable.
16. Normal Curve Equivalents are used because National Percentile Rankings are not interval level data. One of the problems with the transformation from NPRs to NCEs is that the very lowest and highest ends of the distribution are compressed. This tends to inflate very low-end scores and deflate very high-end scores. The lower end inflation may affect this population, which has quite a few test scores below the 10th National Percentile. For later analysis, if sample sizes get large enough, we will also analyze scores by grade using the Iowa Test Standard Scores, which are interval level, but do not have this compression effect. NCEs are, however, the national standard for reporting results across populations and grades for Chapter 1 and other programs.
17. All MPS results in this year's report use the estimated math score described in Technical Appendix F. Last year, we reported the estimated scores in footnotes and appendices. We changed because we feel the more accurate method is to include the estimated scores to prevent an income bias associated with Chapter 1 eligibility. Thus this year we decided to put that version in the text rather than in footnotes and appendices. Table results from last year for 1992-93 changed slightly because of this change in reporting.
18. We are utilizing a more accurate low-income indicator in this year's report. Because many choice students in recent years entered the program in 4 or 5 year old kindergarten, they did not have an established Student Record Data Base in MPS (they never were in MPS). In previous years, their low-income variable was missing data. However, utilizing data on applications and our survey data, we assessed whether those students would qualify for free-lunch or not. Nearly all did because of Choice Program requirements. Thus the low-income variable is now more complete and accurate.
19. See, for example, Robert H. Meyer, "Applied Versus Traditional Mathematics: New Econometric Models of the Contribution of High School Courses to Mathematics Proficiency," Washington, D.C., National Assessment of Vocational Education.
20. See Meyer, op cit.
21. Parental involvement and parental attitude measures are based on the responses to the second wave of surveys conducted in June of each year. Those surveys were sent to all enrollees in the choice program. However, it is impossible to estimate the measurement error based on the attendant respondent bias. It is also difficult to predict, a priori, how it might affect results. Are well-satisfied or unhappy respondents more likely to respond?
22. These numbers are obviously subject to response bias in favor of the program. It is impossible to measure the magnitude of that bias. The approval rate is very high even with a reasonable estimate of bias.
23. Parental choice of schools in MPS, which is extensive, creates a data problem because families enter several choices and there are many last minute switches in schools as waiting lists are cleared, families change their mind, etc. The data entry often does not keep up with this fluctuation. For example, in the last weeks of August and first weeks of September, children may be assigned to schools they never really attend. They may also attend for only a few days when an opening they desire occurs in another school. Also, because much data entry is done on-site, with the following November to determine the percent of moves over the summer. Thus we will underestimate the overall mobility which would include at least some of the moves in September and October as legitimate moves.
24. See John F. Witte, "Private School versus Public School Achievement: Are There Findings That Should Affect the Educational Choice Debate?" Economics of Education Review, 11 (December, 1992): 371-394.
25. We do not have an accurate means of tracking where not selected choice parents enroll their children. Based on responses to our spring surveys, 57% of these students were in MPS schools at the end of the year they attempted to enroll in the Choice Program. Because these parents feel little allegiance to the Choice Program, and because addresses are difficult to obtain for parents who moved during the year, response rates are approximately 30% for these surveys. Obviously, as with attrition students, we miss more students who move out of the area. It is impossible to determine if this indicates a bias in our estimates of those ultimately enrolling in public or private schools, or biases in the other variables of interest.