Children's Scholarship Fund

Our Report
Charity Response

In a nutshell

What do they do? The Children’s Scholarship Fund provides partial scholarships to low-income children in grades K-8, to help them enroll in private schools. These scholarships cover between 25% and 75% of private school tuition (depending on the family's income). The program currently serves more than 25,000 students in the United States, with 5,480 scholarship students in New York City.

Does it work? Having reviewed a variety of academic studies, we do not feel that there is empirical support for CSF's approach of granting partial-tuition scholarships.

What do you get for your dollar? CSF spends approximately $1200 per student, overall; we estimate that the total cost of a child's private-school education (paid for not only by the scholarship, but by the participating family and by the private school's own donors) is comparable to the cost of a public-school education (in the neighborhood of $12,000-$15,000 per student per year in NYC).

The Details

What do they do?

Application and eligibility
Size of scholarships

Does it work?

Studies on academic impact
Impact beyond academic performance
Interpreting the evidence

What does it cost?
The organization
Conclusion
Attachments

A. Application and response
B. Program related attachments
C. Organization related attachments
D. Financials

What do they do?

CSF offers partial-tuition scholarships to low-income students, helping to finance their attendance at private school for grades K-8. The program is open (on a first-come, first-served basis) to all who meet basic income requirements and remain in good standing at their choice of school.

Application and eligibility

CSF awards scholarships on a first-come, first-served basis (Attachment A-1 Pg 3) to students in grades K-8. Families must provide proof of income and enrollment in a private school to be eligible for assistance (Attachment A-1 Pg 3), but there are no academic requirements, and students maintain their scholarships through eighth grade provided that the family continues to pay its portion of tuition and the child maintains good standing at the school s/he attends (Attachment A-1 Pg 4).

Students may attend any accredited school; most attend private and parochial schools (Attachment A-1 Pg 2). We have very little specific information about the children enrolled in CSF and the schools they attend; unlike many of the charities we examine, CSF is relatively "hands-off" (in that its interaction with families is limited to verifying eligibility and providing funding). CSF did provide a list of schools attended by scholarship recipients (Attachment B-13).

Size of scholarships

Awards cover between 25% and 75% of a student’s tuition, and are calculated based on family size and income: families at or below 100% of the federal poverty line receive 75% of their child's tuition, those at up to 185% of the poverty line receive 50% assistance, and those at up to 270% of the poverty line receive 25% assistance (Attachment A-1 Pg 2). In Attachment A-1, CSF New York states that its average scholarship size is $1,488, with an average family share of $1,397 from an average family income of $22,440 (though it also states that the average tuition is $2,915 - we aren't sure why there is an apparent $30 discrepancy).

It seems important to us that scholarship size be carefully calibrated to families' needs. Insufficient scholarships would fail to make private school affordable; on the other hand, over-generous criteria would presumably subsidize families more than necessary, rather than serving as many students as possible. Unfortunately, we have very little to go on in assessing whether scholarship size is well calibrated to income, as CSF did not provide us with the details of how scholarship size is determined. In conversation, CSF's New York Executive Director indicated that CSF monitors the percentage of students that continue in the scholarship program from each year, and seeks to maintain that percentage at 80% or above.

We gain some indication of scholarships' impact on affordability from Attachment B-4, an evaluation of a program similar to CSF's (discussed more below) for which scholarship recipients were chosen by lottery (rather than through a first-come first-served approach). Attachment B-4 Pg 2 states that approximately 75% of scholarship recipients ended up enrolling in private school, compared to 11% of non-recipients.

Does it work?

We find the CSF model to have a great deal of intuitive appeal: anecdotally, we'd guess that the low-income children most likely to be eligible (and interested) have seriously deficient public schools, and would have no trouble identifying private schools with better staff and learning environments. We would therefore expect that children who receive CSF scholarships - allowing them to attend private schools for up to 9 years - would be much better positioned to perform academically. However, from what we've seen, available empirical evidence broadly contradicts this intuition.

CSF and similar programs have been analyzed in a range of studies that awarded scholarships to interested students using randomized lotteries, then followed both the students who had been awarded scholarships and the students who hadn't. There is some dispute over exactly what the results indicate: some researchers point to a small impact on academic performance, found only with African-American students, while others dispute even this impact and argue that the impact of scholarships is essentially negligible. We feel that the latter case is slightly stronger.

We believe there are several possible explanations for why this intuitively appealing program may not have the desired effects, and ultimately would not bet on it as a way of significantly improving students' academic performance.

Studies on academic impact

We reviewed academic research on U.S. voucher programs structured similarly to CSF, including four studies of privately funded voucher programs (in New York, Washington D.C., Charlotte, Dayton) and two publicly funded programs (in Milwaukee and Cleveland). Each study attempted to measure the impact that vouchers or scholarships had on participants' academic performance, as measured by standardized test scores. We found the literature reviews prepared by the RAND Corporation (Attachment B-1 Pgs 79-94) and the Economic Policy Institute (Attachment B-2 Pgs 5-20) particularly informative.

New York City

Researchers followed the academic performance of about 1,300 students who were randomly selected to receive offers of partial-tuition scholarships, as well as 1,300 students who did not receive such offers. All such students entered grades K-4 in 1997; their performance was followed over the course of three years (Attachment B-4 Abstract).

This is the only relevant study we know of for which all raw data is publicly available, making challenges by other researchers possible - and different researchers did reach different interpretations. Specifically:

Paul Peterson and William Howell (then at Harvard) and David Myers and Daniel Mayer (researchers at Mathematica Policy Research) conducted the initial analysis. They found no statistically significant impact on academic performance (as measured by standardized test scores) for the group of students as a whole, but a moderate and statistically significant impact (between 4.1 and 8.2 percentile points on the Iowa Test of Basic Skills) on the academic performance of African-American students (Attachment B-3).
Two researchers at Princeton University, Alan Krueger and Pei Zhu, argued that the apparent impact on African-American students was no longer statistically significant if another group of students (those for whom pre-experiment scores were unavailable) were included in the analysis and the definition of African-American was expanded to include mixed race black/hispanic students (Attachment B-4).
Peterson and Howell responded to Krueger and Zhu, conceding that both proposed adjustments would point to weaker effects, but challenging the validity of these adjustments and observing that even with Krueger and Zhu's definition of African-American, 2 of the 5 classes studied still showed statistically significant improvement on test scores (Attachment B-5).
Reviewing the new analysis presented by Krueger and Zhu, Myers and Mayer concluded that "one must remain cautious when interpreting the findings for African Americans" (Attachment B-6 Pg 5).

We don't find Peterson's and Howell's case very convincing, given both its high sensitivity to the exact methodological approach taken (as Krueger and Zhu demonstrated) and its reliance on after-the-fact decisions to focus on subgroups of the population.

Other experiments

The New York City experiment is the only one we know of that produced a strong data set with few methodological concerns and followup data available for most participants. The other major studies we have investigated are summarized below; we believe their results are inconclusive and add relatively little to our understanding of scholarships' effects.

Howell and Peterson designed and conducted experiments in Washington D.C., Dayton, and Charlotte (Attachment B-8 Pg 6), with similar structure to the New York City experiment. The results they found were broadly consistent with their own findings from New York: no statistically significant impact on the academic performance of the sample as a whole, and moderate statistically significant impacts (around 3-9 percentile points, though a few groups saw smaller or even negative impacts) on the academic performance of African-American students (Attachment B-8 Pg 13, Attachment B-9 Pg 8). However, the quality of these experiments is lower than that of the New York experiment, due to low participation rates in follow-up surveys. 67% of students in the New York experiment participated in follow up exams (Attachment B-3 Pg 43); by contrast, only half of students participated in the follow up exams in Dayton and D.C., and about 40% participated in Charlotte (Attachment B-2 Pg 17). Although we don't know whether students who left the study differed systematically from those who stayed, and thus can't tell whether or in which direction their attrition biased the results, large levels of attrition pose a significant threat of bias to an initially randomized study. This concern is prominent in other comparative discussions of these experiments, including the analysis done by Evidence Based Programs. In addition, because data has not been made public for the non-New York studies, other researchers haven not had the opportunity to critique the results as Krueger and Zhu did for New York City.

Milwaukee conducted a publicly funded voucher experiment. Cecilia Rouse of Princeton and Jay P. Greene of the Manhattan Institute found positive impacts on academic performance for students who received vouchers; John Witte of the University of Wisconsin responded with a critique of their methodology, observing that only 48% of the students lotteried out of the voucher program stayed in the public school system, and there is reason to believe that the remaining group included a disproportionate number of low performing and learning-disabled students (those who could not, or would not, gain admittance to a private school). Witte's re-analysis found a small negative impact of receiving a voucher (Attachment B-2 Pgs 7-8).

We do not discuss the Cleveland experiment (Attachment B-10 Pg 4); this was not a lottery-based experiment, and there is little data on background characteristics of students. It's therefore highly unclear to what extent the two groups of students (recipient and non-recipients of scholarships) were otherwise comparable.

Impact beyond academic performance

Some researchers have attempted to evaluate the impact of voucher/scholarship programs beyond academic achievement, looking at the difference in altruistic behavior by students (and their parents) who are awarded vouchers vs. students and parents who are not. While we are sympathetic to the idea that such non-academic impact is important, the analysis we've seen on this topic - submitted by CSF - does not, to us, offer a compelling case for the impact of scholarships.

The study CSF submitted (Attachment B-14) conducted an exercise in which students were given money and then offered the opportunity to donate to a specified charity; students who chose to donate more were taken to be demonstrating greater altruism. The study found that students who (by lottery) had been awarded scholarships for private schools rated as more altruistic than students who had not been awarded such scholarships (Attachment B-14 Pg 4). The researchers used similar experiments to gauge students' generosity toward their peers, and the generosity of their parents, finding no statistically significant effects for either (Attachment B-14 Pg 4). Furthermore, the two groups of students (those who had been awarded scholarships and those who had not) do not appear otherwise comparable: those who were awarded scholarships had significantly higher incomes (to begin with) than those who were not (Attachment B-14 Pg 30).

Interpreting the evidence

While the idea of providing scholarships to low-income students is intuitively appealing, the following factors may prevent it from significantly improving academic outcomes:

Families who choose to use scholarships may be more motivated than other families to begin with, such that they would find other ways of pursuing a better education even in the absence of these scholarships. This possibility seems especially strong when the scholarships are partial rather than full (as is the case for CSF as well as for the voucher experiments discussed above), meaning that only families willing to pay extra for education participate.
If disadvantaged children are already academically far behind by the time they are eligible for these scholarships, it's possible that a private school - focused largely on serving less disadvantaged children - is not the appropriate remedy. Schools such as those run by KIPP primarily serve disadvantaged children, and may thus be better positioned to build a curriculum around their particular needs and help them catch up.

Most of the empirical evidence we discuss above is highly ambiguous, due to methodological concerns; however, the New York City Voucher Experiment appears to have produced a relatively clean and bias-free data set, and it implies that the impact of scholarships on academic performance is small (and limited to subgroups of the population) if it exists at all.

What does it cost?

CSF expects to spend approximately $30 million to provide scholarships to 25,000 children during the 2008 school year (Attachment D-1). Average tuition at a school attended by a CSF scholarship recipient is about $3,000 (Attachment A-1 Pg 3).

However, the actual cost of educating a student at schools that CSF scholarship recipients attend is likely significantly higher than the tuition, as private schools may rely on fundraising of their own to supplement revenue from tuition. We don’t have data on costs for the schools in which CSF students enroll (these schools are religious institutions and are not required to file their financials publicly), but to get a ballpark figure, we looked at the financials of St. Aloysius, a parochial school in New York City that is also one of our applicants. St. Aloysius expects total expenses of $3.25 million in 2008 and serves approximately 300 students, which comes out to overall expenses of ~$11,000/student. This in the same ballpark as public school per-student costs ($13,755 per student in NY, according to the US Census Bureau).

The organization

Financials. All data is taken from CSF's Form 990s (available via GuideStar), Attachment A-2 Pg 5, and Attachment D-1.

Year	Revenues (in thousands)	Expenses (in thousands)
2004	$8,983	$9,416
2005	$16,592	$15,734
2006	$19,806	$20,466
2007 (exp)	$23,275	$23,279
2008 (exp)	$29,034	$28,829

In 2006, the latest year for which a Form 990 is available, CSF held $8,017,644 in assets (or approximately 40% of that year's expenditures).

Board of directors. CSF has a fifty-member National Board of Advisers that includes businessmen, prominent politicians (from both sides of the aisle), educators, and non-profit leaders. The exact role of the Board in the management of the organization is unclear to us.

Conclusion

On its face, we find the CSF model intuitively compelling; however, the empirical evidence shows much weaker effects than we would expect. Ultimately, we have more confidence in KIPP, whose fundamentally different approach is supported by what we feel is a stronger (although less rigorous) empirical case.

Attachments

A. Application and response

Attachment A-1: Round 1 Application Response
Attachment A-2: Application - Round 2
Attachment A-3: Round 2 Application Response

B. Program related attachments

Attachment B-1: RAND School Voucher Literature Review
Attachment B-2: EPI School Voucher Literature Review
Attachment B-3: “School Choice in New York City” – Howell and Peterson (2002)
Attachment B-4: “Another Look at the New York City Voucher Experiment” – Krueger and Zhu (2003)
Attachment B-5: “Efficiency, Bias, and Classification Schemes” – Peterson and Howell (2003)
Attachment B-6: “Comments on ‘Another Look at the New York City Voucher Experiment’” – Myers and Mayer/Mathematica Research (2003)
Attachment B-7: “An Evaluation of the Children’s Scholarship Fund” – Peterson (2001)
Attachment B-8: “Test Score Effects of School Vouchers in Dayton, New York, and Washington D.C.” – Howell, Wolf, Peterson, and Campbell (2000)
Attachment B-9: “The Effect of School Choice: An Evaluation of the Charlotte Children’s Scholarship Fund” – Greene (2000)
Attachment B-10: Jay P Greene Literature Review (2000)
Attachment B-11: Complete List of CSF studies
Attachment B-12: “Baseline Findings for an Evaluation of the Children’s Scholarship Fund(CSF) in Los Angeles” – Quigley (2006)March8-06FINALQuigley
Attachment B-13: List of Public Schools From Which Scholarship Recipients Came and Private Schools In Which They Enrolled
Attachment B-14: Bettinger, Eric and Robert Slonim. (2004). Using Experimental Economics to Measure the Effects of a Natural Educational Experiment on Altruism. NBER Working Paper 11725. [CSF submitted this in hardcopy. No ungated copy available.]

C. Organization related attachments

D. Financials

Attachment D-1: Budget For Fiscal Year 2008
Attachment D-2: Expected Major Donations in 2008 (withheld due to confidentiality request)

Charity Response

(Edit)

In a Nutshell
• The number of children you quote as receiving scholarships is dated from 2006-07. In the 2007-08 school year, almost 29,000 children nationwide used CSF scholarships. In New York City, we served more than 8,800 recipients.
• The actual cost to educate K-8 children in the schools where our recipients attend is significantly lower than the $12,000-$15,000 amount you note that NYC spends on public school students. The average cost in the Catholic schools of the Archdiocese of New York – where approximately more than 6,000 of our NYC recipients attend – is $3,617.

What Do They Do
• While it is true that CSF has no academic requirements for its applicants, the schools that they apply to do have their own admissions testing and standards. Children entering the schools must go through the same process any other applicant would undergo, and we trust the schools’ established placement procedures.
• Rather than being “hands-off” with our recipients, CSF has many points of contact with them throughout the school year. During the application and requalification periods, many families deliver their paperwork in person to our offices. We also phone each family up to three times each summer and mail up to three letters to follow up on paperwork, and families are required to come into the school at least twice a year to sign verification reports. Families and schools also receive regular mailings from CSF during the school year, including a packet for 8th graders advising them of high school scholarship programs and the high school enrollment process. In fact, we have a very collaborative partnership with the schools, which includes weekly contact with school principals, who of course have direct contact with the families. With the principals, we are able to shepherd families through the application process and beyond into the culture of a private school. It is difficult to measure, but choosing a school and then paying part of the tuition has a very definite effect on the entire family.
• Regarding the calculation of our scholarship amounts, the formula we use was thoughtfully developed, carefully calibrated, and very straightforward. Using the same guidelines as the Federal Free and Reduced School Lunch program, we take into consideration the number of people in the household, the total family income, and the tuition at the school. The family receives either 25%, 50%, or 75% of the tuition charged, up to a maximum of $2,400, because, based on these factors, that is the amount we have determined is necessary for the parent to make the decision to choose a private school. All families must pay at least $500 a year as a minimum investment in their child’s education because it is important that they are a partner in this effort. (In NYC in 2006-07, the average award was $1,488, the average family paid $1,397, and the average tuition was $2,915. Financial aid from the school and other sources makes up for the $30 difference between the sum of the scholarship and family contribution and the average tuition size.) We have a high level of confidence in our formula. We monitor the number of families who are in the 75% tuition category. If too high a percentage of our recipients are in this category, we realize that the scholarship amount needs to be increased. For this reason, we reconsider the maximum scholarship amount each year, and for 2008-09 the amount will be $2,400, $300 higher than the $2,100 cap for 2007-08. As noted above, it has allowed families with the necessary amount of assistance to choose a private school, while ensuring that they themselves have some stake in the process. The best indicator that the formula works is the fact that our retention rate is over 80% - meaning that less than 20% of our recipients choose to leave the program. Based on Exit Confirmation Forms that we receive from the parent, most of the families who leave do so because they are moving to an area outside the program.

Does it Work?
• CSF had more than 8,800 children using scholarships in New York City in 2007-08, and we do not compile grades and test scores for all of these children, although we encourage researchers to undertake studies of our recipients, and we cooperate with them to do everything possible to facilitate such studies. To provide you with an example of data that is available for schools our participants attend, we asked the Archdiocese of New York to provide test scores for a couple of Catholic schools with large numbers of scholarship recipients and then compared them to the public school scores for the school districts they lie in.
• For example, at St. Nicholas of Tolentine School in the Bronx, 122 children out of the school’s total enrollment of 377 (32.3% of the student body) are using CSF scholarships to attend. This year, 100% of 4th graders met or exceeded NY state standards in English Language Arts, compared to only 54% of District 10 4th graders. In math, 97% of St. Nicholas 4th graders met or exceeded the state standard, compared to 72.8% of district children.
• At Mt. Carmel-Holy Rosary School in East Harlem, almost half of the school (121 of 253 children) uses CSF scholarships. In 2008, 97% of the school’s 4th graders met or exceeded state standards in English Language Arts, while only 54% of 4th graders in District 4 public schools met the same standards. 100% of Mt. Carmel-Holy Rosary 4th graders met or exceeded the state math standards, compared to 74.7% of the local public school 4th graders.
• While the studies linked to show gains that scholarship recipients have made upon attending private schools, we consider parental satisfaction to be another clear measure of success. Although your analysis does not address it, the 2001 Harvard study, “An Evaluation of the Children's Scholarship Fund,” (Attachment B-7) showed that 68% of CSF families indicated they were “very satisfied” with their chosen private schools’ academic quality, safety, discipline, and the values taught in the school. Only 23% of their public school counterparts felt similarly. CSF parents were approximately five times more likely to rate their school an “A” than public school parents—72% versus 16% respectively.

A Wall Street Journal editorial applauded the study and ended, “Now, skeptics can debate the findings and point out that the study does not measure performance. But surely that would be to miss the point. For what makes this study so distinctive is not just what it discloses about parental satisfaction. What makes it distinctive is that it treats parental satisfaction as an important question.” The editorial also noted that, while it may not be surprising that private school parents would rate their schools highly, scholarship parents who are sacrificing a significant portion of their low-income (in 2007-08 the average NYC scholarship family paid $1,390 out of a total income of $22,630) to contribute to tuition are likely to be more demanding than others (“Private Matters,” Wall Street Journal, Friday, May 4, 2001).

What Does It Cost?
• St. Aloysius School’s cost per student as you have calculated it is not indicative of the average cost for New York City inner-city Catholic schools. According to the Archdiocese of New York, the average cost for educating an elementary student in their inner-city schools is: $3,617.
• Because donors are effectively partnering with CSF and the scholarship families to pay for tuition, their contributions are highly leveraged. Charity Navigator has awarded CSF a four-star rating – its highest possible rating – for financial efficiency.

United States