The bottom line for donors

In reviewing the evidence for education programs, we observe that:

• Little reliable information is available regarding how to improve school attendance and school quality, as well as on the true relationship between schooling and later-life outcomes such as income. Improving developing-world education is not a matter of getting proven programs to those who can't afford them.
• What information is available illustrates that many logical-seeming and generous interventions (such as providing textbooks/supplies) have often failed to produce results. There are also programs that have had strong results, but evidence is extremely limited, and we would expect similar programs to have different effects in different areas and contexts.

If you're committed to the cause of education, we recommend that you investigate the following questions for any charity you're considering:

• Is the goal to improve attendance or to improve school quality?
  • If the goal is to improve school attendance, what sorts of schools will beneficiaries be attending? What information is available on teacher attendance and quality of instruction? (We also a recommend a visit to a representative school if possible.)
  • If the goal is to improve school quality, what particular skills are being emphasized? Is there a process in place to gauge children's progress on these skills?
• What is the evidence that the skills being taught are (a) realistic for the children at the school; (b) likely to be useful as the children grow older? What are the economic opportunities that will be available to students and how do education levels relate to them?

You may want to consider one charity, Pratham.1 (For more, see our brief review of Pratham.) Pratham has, in the past, shown a commitment to rigorous evaluation of their programming. This commitment does not by itself answer all the questions above, but may indicate an organizational commitment to learning about what works and holding themselves accountable. This charity has been closely involved with some of the studies discussed below2 and has completed a number of projects that have been evaluated by the Poverty Action Lab at M.I.T.3

Summary of the evidence

Here we review available information about programs seeking to improve school-based education in the developing world. We rely mostly on a recent review of literature on the subject.4

We feel that:

• Programs seeking to build schools and provide scholarships - common among charities - may successfully raise school attendance.
• The value of raising school attendance is questionable given evidence about the poor quality of school systems in the developing world.
• Evidence about how to improve the quality of education is quite limited. Some programs appear to have succeeded in raising test scores; others, including textbook/classroom materials provision, appear to have failed to have done so.
• The connection between school-taught skills and life outcomes is extremely unclear.

Details

Improving attendance: school building and scholarships

Many charities focus on building schools and/or providing scholarships to offset private school fees (which are common in the developing world5). Charities we examined that focus exclusively on these activities include:

• AWaPo
  
• OneKid OneWorld
  
• Orphans of Rwanda
  
• Root Foundation
  
• CO-ID

A major concern with programs like these is that they may accomplish little if the schools are not well-run, and available research implies that the quality of schooling in the developing world is often extremely poor (well beyond what is considered "poor" in the U.S.)

• Teachers are frequently underqualified, overworked, and/or frequently absent.6
• In some cases teachers have even been observed to be abusive to both students and parents (the latter by pressuring them to pay for extra lessons).7
• Schools are often geared specifically toward elite students, providing little or no benefit to students who aren't prepared for the curriculum (or even in some cases the language of instruction).8

School building

Research on school building is slightly encouraging, but relatively sparse and confined to a few specific places and times. In order to have confidence in a charity focused on school building, we would need a solid understanding of the communities in which the schools are being built (in particular, who is likely to staff them and what curriculum and system they will be under - see above).

• Studies in Ghana, Tanzania, and India find associations between people's years of school completed and various measures of their schools' accessibility/building quality (measures include distance/travel time to the nearest school; supply of blackboards; functional roofs and "waterproof" classrooms). However, these studies are plagued by a number of concerns, from bad data to questions about causality (is it the school quality that creates the additional attendance, or are other factors - perhaps higher-income or more education-focused communities - responsible for both attendance and quality?)9 In addition, these studies don't address the concerns about school quality raised above.

• One study examines an extremely large-scale government project in Indonesia that built a large number of schools in the 1970s.10 Because only people of a certain age and living in certain regions were able to attend the new schools, the study compares these people to those who could not attend, due to their year of birth and/or geographical location.11 This is a more rigorous study than the others, because it appears from available information that differences between such groups can reasonably be attributed to construction of schools themselves.12 The study's results suggested that individuals who benefited from the program received slightly more years of education (0.12 years on average) and had slightly (2.6%) higher wages about twenty years after participating in the program; these apparent effects were not statistically significantly different from zero.13

Scholarships

Studies from Bangladesh and Pakistan show mixed impacts of scholarships on school enrollment/completion (the Pakistan study concludes success for urban areas and failure for rural areas; the Bangladesh study concludes the program was successful in raising enrollment rates). They do not examine impact beyond enrollment/attendance and may not be correctly identifying the effect of the scholarship programs, as opposed to other changes taking place at the same time.14 A merit scholarship program (discussed below) was found to have encouraging effects.

As with school building, we would need substantial information about a community and school system to feel confident in a scholarship program.

Another concern regarding scholarships is that they may lead to increased class sizes, which could degrade the quality of education (see "Improving teacher:pupil ratios" below).

There is relatively little strong evidence regarding what sorts of programs have been successful in the past (in terms of raising graduation, test scores, and/or later life outcomes). As with U.S. education, many studies have significant flaws due to selection bias and other concerns.15 Here we summarize the results from stronger studies of improving developing-world schooling. Many logical-seeming programs, including provision of textbooks/supplies, have had mixed or no effects.

Textbooks and supplies

The very limited existing evidence implies that providing school supplies is not, by itself, key to improving school quality.

• A study in Kenya randomly assigned some schools to receive free textbooks (official Kenyan government textbooks). The authors found no aggregate effect on test scores, although students who had strong performance to begin with did improve. The authors note that the language and content of the textbooks may have been inappropriate for the students.16

• The same organization that had experimented with textbooks then tried "flip charts" (poster-sized charts with instructional material) in the hopes that they would be more appropriate.17 Although initial studies implied that the flip charts had led to improvement, a rigorous study (based on random assignment of classrooms to receive them) showed no impact.18
• A program in Nicaragua randomly selected 48 classrooms to receive radio-based instruction in mathematics, 20 to receive mathematics workbooks, and 20 to serve as a comparison group (receiving neither). Those with workbooks scored higher on mathematics tests after a year than those without; those with radio instruction outperformed both other groups by a substantial (and statistically significant) amount.19

• A study in Kenya provided randomly selected schools with uniforms, textbooks, and classrooms. The program appeared to lower dropout rates and increase completion rates, though it did not have any apparent impact on test scores. The authors argue that its main effect was attributable to the provision of uniforms (as opposed to textbooks and classrooms).20 The program also led to significantly increased class sizes,21 which may have offset other benefits (more below).

Improving teacher:pupil ratios

Some programs seek to hire more teachers in order to lower class sizes. Past examinations of this approach have found positive effects on enrollment/completion and mixed effects on test scores.

• A government program in South Africa increased funding for schools in a way that led to uneven changes in class sizes, changes that were interpreted and studied as a "natural experiment" on the impact of class size. The study concluded that smaller class sizes did lead to more years of school completed, for blacks but not for whites (possibly because blacks had much larger class sizes to begin with). Test scores were examined as well, but no effect was found. This study may have suffered from selection bias, depending on the details of how funding was allocated.22

• A rigorous evaluation in India examined a program that "attempted to raise school quality by hiring additional teachers, especially female teachers" in randomly selected nonformal schools (focused on "basic numeracy and literacy schools"). The evaluation found that
  

  The program reduced the number of days a school was closed (one-teacher schools were closed 44 percent of the time, whereas two-teacher schools were closed 39 percent of the time), and girls’ attendance increased by 50 percent. However, the program had no significant effect on the attendance of boys.

  Impact on literacy/numeracy as measured by test scores was not examined. 23

Teacher incentives

A rigorous evaluation of a "pay-for-performance" program in Kenya found discouraging results implying that it mostly led to "teaching to the test." Schools were randomly selected to participate in the program, which "offered teachers prizes based on their schools' average scores on district-wide exams" (while penalizing them for dropouts). The program led to improved test scores in the short term, but the largest improvements were on the most memorization-dependent tests, and no lasting improvement was discernible a year after the program ended. It appeared that teachers responded to the program largely by conducting outside-of-school test prep sessions.24

Computer- and radio-assisted learning

• A more rigorous evaluation occurred in India, randomly assigning some schools to participate in a nonprofit organization's computer-assisted learning program. Scores on math tests improved for participants in both the first and second years of the program.25

• A superficially similar program in Colombia – funded by private-sector donors – had less encouraging results. A two-year randomized evaluation concluded that "Overall, the program seems to have had little effect on students' test scores and other outcomes" and that "The main reason for these results seems to be the failure to incorporate the computers into the educational process."26

• A program in Nicaragua found encouraging results from radio-assisted learning (this study is discussed above).

Other programs

Randomized evaluations have found encouraging effects for four programs that appear to us to be relatively rare among charities. Two of these programs are focused on health/nutrition rather than pedagogy.

• A program in Kenya introduced free breakfast to a randomly selected set of 25 preschools (with 25 serving as a comparison group.) Preschools with free breakfast had 30% higher attendance than those without, and some limited evidence implied higher scores on academic tests (although this effect was only found for part of the sample and is therefore particularly questionable). 27

• An evaluation in rural Kenya randomly selected schools to participate in the Girl's Scholarship Program, "a merit-based scholarship awarded to girls in two districts of Western Kenya who scored in the top 15 percent on tests administered by the Kenyan government." Schools participating in this "pay-for-performance for students" program saw improved test scores (even for girls with low prior test scores, whom the authors conjecture were unlikely to win scholarships, as well as for boys, who were ineligible) as well as improved teacher attendance.28

• A remedial education program in India employed local volunteers to help struggling children with their reading skills. The program, run as a collaboration between the Indian government and a nonprofit, was rolled out randomly, allowing a clean comparison between participants and non-participants. The comparison found that participants had superior test scores in both the first and second year.29

• A deworming program – mass administration of drugs to eliminate parasites – was found to raise attendance at pseudo-randomly selected schools in Kenya.30 Full details of this study (and other studies that are argued to be less meaningful) available here.

Beyond test scores

All of the studies discussed above focus on very short-term impacts on results such as school attendance, completion, and test scores. However, a major question of ours is whether – and if so, how – children in developing countries ultimately benefit from the skills taught in school.

Higher-education enrollment rates in South Asia and Africa are extremely low (under 10% for the regions as a whole, and possibly lower for the areas most likely to be focused on by charities),31 and a recent review states that "little is known about what is happening within universities and to the students who spend a considerable part of their prime years there."32 There may be many economic opportunities open to people in the developing world who can read, do arithmetic, etc.; there may not be; or it may depend on the region.

A large number of studies have linked education levels to earnings levels in the developing world, but nearly all of them suffer from extremely serious selection bias issues, since those who complete more years of education are likely to have many advantages over those who don't (for example, higher income and more favorable locations). Three relatively recent papers on this topic note this problem with past research, while attempting to draw more valid conclusions about the connection between education and earnings; we find all three problematic, but even if one accepted their conclusions at face value, they show mixed results in a limited number of regions.

• A 2002 paper notes that "most studies on this topic have important limitations, particularly in studies for developing countries. They tend to ignore behavioral decisions regarding schooling and individual and family background characteristics, use school quality measures aggregated to the regional level, and rely on crude indicators of teacher quality."33It presents its own analysis from rural Pakistan, separately estimating the relationship wage earnings and cognitive skills (as measured by tests of literacy and numeracy)34 and the relationship between cognitive skills and schooling (while attempting to adjust for a variety of background characteristics including innate cognitive ability).35 Its analysis finds positive relationships between school quality and cognitive ability36 and between cognitive ability and wage earnings.37 We find the analysis problematic in a variety of ways.38

• A 2002 review states that most past research has focused on the relationship of earnings to years of schooling and earnings, whereas examining the relationship to cognitive skills (such as reading and arithmetic) would allow better disentangling of the effects of learning vs. other factors including "sheepskin effects" (i.e., the signaling value of a diploma, regardless of whether it's associated with learning).39 It focuses on studies that have examined the relationship between cognitive skills and earnings while statistically adjusting for "innate cognitive ability" (as measured by the Raven's Coloured Progressive Matrices Test.
  • It cites positive associations between earnings and cognitive skills (after an adjustment for "innate cognitive ability") in three studies: one on urban wage earners in Kenya and Tanzania, another on wage earners in Ghana, and another on wage earners in rural Pakistan.40

  • It notes that "these studies focus on wage workers even though self-employment is more common in all of these countries … Regrettably, only two published studies have examined the impact of cognitive skills on self-employment income in developing countries." Both use data from Ghana; one is reported to find positive effects overall, but not for agricultural workers, while the other finds "only weak evidence" for a relationship between cognitive skills and earnings. Possible selection bias problems are noted for both.41

  We find this analysis to have similar problems to the study cited directly above.

• Another relevant study is the one cited above on the rapid school construction project in India, finding positive but not statistically significant effects of school availability on wages. Like the others, this study notes that the large amount of past research is not reliable.42

Sources

• Banerjee, Abhijit, Shawn Cole, Esther Duflo and Leigh Linden. 2005. "Remedying Education: Evidence from Two Randomized Experiments in India." BREAD Working Paper 109. Available online at http://www.povertyactionlab.org/papers/4.pdf, accessed 7/7/09.
• Barrera-Osorio, Felipe and Linden L. Leigh. 2009. "The use and misuse of computers in education: evidence from a randomized experiment in Colombia." World Bank. Available online at http://econ.worldbank.org/external/default/main?pagePK=64165259&piPK=641..., accessed 7/7/09.
• Behrman, Jere, David Ross and Richard Sabot. 2002. "Improving the Quality Versus Increasing the Quantity of Schooling." PIER Working Paper 02-022. Available online at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=322200, accessed 7/7/09.
• Duflo, Esther. 2001. "Schooling and Labor Market Consequences of School Construction in Indonesia: Evidence from an Unusual Policy Experiment." American Economic Review 9:4. Available online at http://www.pse.ens.fr/gurgand/Duflo01.pdf, accessed 7/7/09.
• Glewwe, Paul. 2002. "Schools and Skills in Developing Countries: Education Policies and Socioeconomic Outcomes." Journal of Economic Literature, 40(2): 436-482.
• Glewwe, Paul and Michael Kremer. 2005. "Schools, Teachers, and Education Outcomes in Developing Countries." Forthcoming in Handbook on the Economics of Education, Elsevier. Available online at http://www.economics.harvard.edu/faculty/kremer/files/EconEducationHandb..., accessed 7/7/09.
• Kapur, Devesh, and Megan Crowley. 2008. "Beyond the ABCs: Higher Education and Developing Countries." Center for Global Development Working Paper. Available online at http://www.cgdev.org/content/publications/detail/15310, accessed 7/7/09.
• Kremer, Michael. 2003. "Randomized Evaluations of Educational Programs in Developing Countries: Some Lessons." American Economic Review 2003. Available online at http://www.economics.harvard.edu/faculty/kremer/files/Randomized_Evaluat..., accessed 7/7/09.
• Miguel, E., and M. Kremer. 2004. "Worms: Identifying impacts on education and health in the presence of treatment externalities." Econometrica 72(1):159–217. Available online at "http://weber.ucsd.edu/~tkousser/Miguel and Kremer.pdf", accessed 7/7/09.
• Poverty Action Lab. Organization website. Online at http://www.povertyactionlab.com/, accessed 7/9/09.

1. 1.

   http://www.pratham.org/, accessed 7/7/09.

2. 2.

   "Nongovernmental organizations in developing countries may be very well placed to conduct randomized evaluations. Unlike governments, NGOs are not expected to serve entire populations. Also unlike governments, financial and administrative constraints often lead NGOs to phase in programs over time, and randomization will often be the fairest way to of determining the phase-in order. In contrast to developed countries, where NGOs typically do not have sufficient resources to conduct large programs that could serve as a model for public policy, this is not the case in developing countries. Since many NGOs exist and they frequently seek out new projects, NGOs willing to conduct randomized evaluations can often be found. For example, the set of recent studies conducted in Kenya have been carried out through a collaboration with the Kenyan NGO Internationaal Christelijk Steunfonds (ICS) Africa. ICS was keenly interested in using randomized evaluations to see the impact of its programs as well in sharing credible evaluation results with other stakeholders and policymakers. A second example is the collaboration between the Indian NGO Pratham and researchers from the Massachusetts Institute of Technology that led to the evaluations of the remedial education (Banerjee 2000) and computer-assisted learning programs (Banerjee 2004). However, while NGOs are well placed to conduct randomized evaluations, expecting them to finance the research is less reasonable, as the results are global public goods. The evaluations of the ICS deworming programs were made possible by financial support from the World Bank, the Partnership for Child Development, the U.S. National Institutes of Health, and the MacArthur Foundation. In the case of the Indian educational programs, Pratham found a corporate sponsor, India’s second-largest bank, ICICI Bank, which was keenly interested in evaluating the impact of the program and helped finance part of the evaluation." Glewwe 2005, Pg 53.

3. 3.

   See http://www.povertyactionlab.org/getinvolved/?type=4, accessed 7/7/09.

   Two other J-PAL parterns, ICS Africa (http://www.ics.nl/index.php?id=22, accessed 7/7/09) and Seva Mandir (http://www.sevamandir.org/, accessed 7/7/2009) have also had some education projects reviewed by J-PAL. Donors should note that these organizations have a broader scope than Pratham, implementing projects including health, economic empowerment and other activities.

4. 4.

   The review is Glewwe 2005. This paper criticizes the "retrospective study" methodology on pages 14-17, and favors the "natural experiment/randomized trial" methodology on pages 17-18. In general, we address all the studies it includes in the latter category, with the exception of three studies that were done on data from Israel (country-wide) in the 1990s; we don't feel this environment is appropriately included in the "developing-world" category we focus on.

5. 5.

   "This low spending on primary and (to a lesser extent) secondary education in developing countries often implies that households bear much of the cost of that education. Thus parents, rather than the school or ministry of education, are responsible for providing many basic school inputs such as textbooks, chairs, and even the school building itself. Some of these costs are the collective responsibility of parents, but some are passed on to parents through official or unofficial school fees or by requiring parents to purchase uniforms and textbooks for their children. Data on such costs are not available for many countries, but a few examples are worth considering, although it is worth bearing in mind that they may not be representative. In Jamaica, government expenditures per primary school student are US$221 while private expenditures are $178 (Planning Institute of Jamaica, 1992). In the Philippines, the analogous figures are $110 and $309 (Asian Development Bank, 1999), and for Vietnam they are $23 and $14 (World Bank, 1997b). These figures include private school costs." Glewwe 2005, Pg 9.

6. 6.

   "the quality of schooling in developing countries is often very low. Grade repetition and leaving school at an early age are common, teachers are often absent from classrooms, and many children learn much less than the learning objectives set in the official curriculum (Lockheed and Verspoor, 1991; Harbison and Hanushek, 1992; Hanushek, 1995; Glewwe, 1999). Visitors from developed countries are often shocked at the conditions in many (but not all) schools in developing countries. Many schools lack the most basic equipment and school supplies—textbooks, blackboards, desks, benches, and sometimes even classrooms (in which case classes meet outside and are cancelled when it rains). In rural areas of Vietnam’s Northern Uplands region in 1998, 39 percent of primary school classrooms did not have blackboards. In India in 1987, more than 8 percent of schools did not have a building in which to meet (World Bank, 1997). Teacher quality and availability is also a common problem. In rural areas of Northeast Brazil in the early 1980s, 60 percent of primary school teachers had not even completed primary education (Harbison and Hanushek, 1992). Shortages of teachers and school buildings can result in double shifts (which shorten the school day for individual pupils) or very large class sizes. In Vietnam, more than 90 percent of children in rural areas attend schools with two or more shifts, resulting in an average class time of only 3 hours and 10 minutes per day (Glewwe, 2004). In districts with low literacy rates in the Indian State of Tamil Nadu, the average class size in primary school was 78 students (World Bank, 1997). Teachers often have weak incentives and little supervision, and their absenteeism runs high. Chaudhury et al. (forthcoming) reports that when enumerators made surprise visits to primary schools in six developing countries, on average (across these countries) about 19 percent of teachers were absent. Beyond absence, many “present” teachers were found to not be actually teaching; for example, in India one quarter of government primary school teachers were absent from school, but only about half of the teachers were actually teaching in their classrooms when enumerators arrived at the schools." Glewwe 2005, Pg 2.

7. 7.

   "A team of researchers who visited schools in India (PROBE Team, 1999) found some teachers who kept schools closed or nonfunctional for weeks or months at a time, drunken teachers, and a headmaster who expected the students to perform domestic chores and babysit. Sexual abuse of female students by teachers is a problem in several countries. To the extent that teachers do have incentives, these incentives are often focused on exam scores. Teachers often instruct by rote, sometimes copying from textbooks onto the blackboard and having students copy from the blackboard onto notebooks or slates.

   The lack of teacher accountability in many developing countries may reflect the colonial legacy, the hierarchical nature of many developing societies, and the large gaps in education and social status between teachers and their pupils’ families. In many countries, teachers offer, and pressure parents to pay for, “extra lessons” after school or on weekends to prepare students for important examinations (Bray, 1999). In such situations, increased teaching effort at school could reduce the demand for extra lessons, and thus teacher income." Glewwe 2005, Pg 10.

8. 8.

   "Another unusual characteristic of many developing countries is that students are taught in a language that is not their mother tongue. This primarily reflects the fact that almost all developing countries were once colonies of developed countries, and their school systems still embody many elements of the systems developed under colonial rule. Many Sub-Saharan African countries use English or French as their national language, and most of India’s 1 billion inhabits are not native speakers of either of the two official national languages (Hindi and English)." Glewwe and Kremer 2005, Pg 10.

   "Given the heterogeneity in educational background, school quality, and language within many developing countries, designing a single curriculum appropriate for all students is difficult for any country. Yet most developing countries have a single centrally set curriculum, often geared to the needs of relatively elite students, which leaves many other students behind. This contributes to the poor performance of a significant percentage of students on national examinations and to high dropout and repetition rates. For example, in Tanzania between 1997 and 2001, only 22 percent of the students who attempted were able to pass the primary education final examination, and only 28 percent of those who attempted passed the certificate of secondary education exam (Tanzania Media Monitoring, 2002)." Glewwe 2005, Pg 10.

9. 9.

   "A retrospective study in Ghana by Glewwe and Jacoby (1994) presents evidence on the impact of distance and school quality on the years of schooling of individuals aged 11 to 20, using data collected in 1988-89 on household, school, and teacher characteristics. To estimate the impact of school characteristics and other factors on years of schooling attained, an ordered probit specification was used that allows for right censoring. According to the study findings, years of schooling was reasonably responsive to school quality. The estimates indicate that years of completed schooling could increase by 2 to 2.5 years by raising average teacher experience (from 2 years to 10 years), repairing leaking roofs, reducing travel time (from 2 hours to a few minutes), or providing blackboards to schools without them. Since repairing roofs and providing blackboards is much less expensive than building new schools, these results suggest that repairing classrooms in Ghana is a more cost-effective means of increasing the quantity of schooling than building new schools to reduce travel time.

   Although the results from the Ghana study appear plausible, the estimates could be biased for a number of reasons. The data had 18 school and teacher variables, but schools can differ in many more ways, which raises the problem of omitted variable bias. Measurement error in these variables is also a potential problem, either because the assumption that they change little over time is false or because errors were made in collecting the data. Finally, no attempt was made to avoid bias due to endogenous program placement.

   In a retrospective study in Tanzania, Bommier and Lambert (2000) found that distance to school had a significantly negative effect on years of schooling, while the quality of Swahili teaching had a positive effect. However, the authors note some problems with measurement error in these variables. For example, many households reported implausible distances to the nearest primary schools, sometimes more than 100 kilometers. Moreover, since school characteristics were averaged over responses given by households, there could be systematic bias. For example, parents may 'justify' a decision to allow a child to drop out of school by claiming that the local school was of low quality (in the Ghana study school quality variables were collected from schools, not households). Finally, given that there are only four school quality variables, there are serious concerns of omitted variable bias." Glewwe 2005, Pg 21-22.

10. 10.

    "Between 1973–1974 and 1978–1979, 61,807 new schools were constructed (Table 1, panel B), at a cost of over 500 million 1990 U.S. dollars (1.5 percent of the Indonesian GDP in 1973). This represented more than one school per 500 children aged 5 to 14 in 1971, which reportedly makes INPRES the fastest primary school construction program ever undertaken in the world (World Bank, 1990)." Duflo 2001, Pg 797.

11. 11.

    Full methodology laid out in Duflo 2001, Pgs 797-799.

12. 12.

    "The identification assumption should not be taken for granted: The pattern of increase in education could vary systematically across regions. In particular, there could be mean reversion. However, an implication of the identification assumption can be tested because individuals aged 12 or older in 1974 were not exposed to the program. The increase in education between cohorts in this age-group should not differ systematically across regions. In Table 3, panel B, I present this control experiment. I consider a cohort aged 18 to 24 in 1974 and a cohort aged 12 to 17 in 1974. The estimated differences in differences are very close to 0. These results provide some suggestive evidence that the differences in differences are not driven by inappropriate identification assumptions, although they are imprecisely estimated." Duflo 2001, Pgs 798-799.

13. 13.

    "An individual young enough, born in a high program region, received on average 0.12 more years of education, and the logarithm of his wage in 1995 was 0.026 higher. These differences in differences are not significantly different from 0." Duflo 2001, Pg 798.

14. 14.

    "Research on girls’ scholarship programs is limited but suggests that scholarships can have major impacts on girls’ enrollment rates. Research on a small fellowship program in Pakistan that subsidized girls’ primary education in private schools was shown to be successful in urban areas but a failure in rural areas (Kim, Alderman, and Orazem, 1999; Alderman, Kim, and Orazem, 2003). A national scholarship program for girls in rural Bangladesh increased girls’ enrollment rates even after controlling for other measurable influences (World Bank, 2001b). Because with economic development enrollment of girls usually rises (and the gender gap between boys’ and girls’ enrollments narrows), it is potentially very problematic to draw conclusions from before and after comparisons of girls’ enrollment rates. This difficulty highlights the need for randomized evaluations of such programs." Glewwe 2005, 29.

15. 15.

    See Glewwe and Kremer, Pgs 29-34 for an overview.

16. 16.

    "Glewwe, Kremer, and Sylvie Moulin (2003) find no evidence that provision of official Kenyan government textbooks increased scores for the typical student. However, they do find evidence that textbooks led to higher test scores for the subset of students who scored well on a pretest. The authors note that English, the medium of instruction in Kenyan schools and the language in which textbooks were written, was the third language for most pupils, and cite evidence that many pupils had difficulty reading the books. As discussed further below, there is reason to think that the Kenyan curriculum is not appropriate for the typical student in rural areas." Glewwe and Kremer 2005, Pg 38. (The study itself confirms that primary schools were chosen at random.)

17. 17.

    "Given the results with textbooks, the NGO tried providing an alternative input, flip charts that presumably were more accessible to weak pupils." Kremer 2003, Pg 5.

18. 18.

    "A third Kenyan study, not discussed above, is Glewwe and others (2004). It examined flip charts: large poster-sized charts with instructional material that can be mounted on walls or placed on easels. This intervention, which was not examined in Section IV because it did not evaluate the impact of flip charts on any indicators of the quantity of schooling, covered 178 primary schools, half of them randomly selected to receive flip charts covering science, mathematics, geography, and health. Despite a large sample size and 2 years of follow-up data, the estimated impact of flip charts on student test scores is very close to zero and completely insignificant. In contrast, several conventional OLS estimates, which may suffer from many of the problems described in subsection III.B, show impacts as large as 0.2 standard deviations, 5 to 10 times larger than the estimates based on randomized trials." Glewwe and Kremer 2005, Pg 38. (The study itself confirms that were chosen at random.)

19. 19.

    "Jamison and others (1981) conducted a randomized trial in Nicaragua in which 48 first-grade classrooms received radio mathematics instruction, 20 received mathematics workbooks, and 20 served as a comparison group. After 1 year, on mathematics tests the radio students scored more than one standard deviation higher, and the workbook students about a third of a standard deviation higher, than students in the control group. Both differences were highly statistically significant." Glewwe and Kremer 2005, Pg 37.

20. 20.

    "Kremer and others (2002) conducted a randomized trial in rural Kenya to evaluate a program in which a nongovernmental organization (NGO), Internationaal Christelijk Steunfonds Africa (ICS), provided uniforms and textbooks and built classrooms for 7 schools randomly selected from a pool of 14 poorly performing schools. Dropout rates fell considerably in the 7 schools selected for participation, and after 5 years pupils in those schools had completed about 15 percent more years of schooling. In addition, many students from nearby schools transferred into program schools, raising class size by 50 percent. This suggests that students and parents were willing to trade off much larger class sizes for the benefit of free uniforms, textbooks, and improved classrooms. The authors argue that the main reason for the increase in years of schooling is most likely the financial benefit of free uniforms. A randomized trial of textbook provision in Kenya, discussed in the next subsection, showed almost no impact of textbooks on the quantity of schooling, and while the new classroom construction may have had an impact, the first new classrooms were not built until the second year of the program, while dropout rates fell dramatically in the first year. Anticipation of later classroom construction may have influenced these results, but the authors doubt it, because effects were present for students in the upper grades who would have finished school by the time the classrooms were built." Glewwe and Kremer 2005, Pg 24.

21. 21.

    "The program schools attracted a large influx of pupils from neighboring schools, increasing average class size by 8.9 students." Glewwe and Kremer 2005, Pg 47.

22. 22.
    • "Case and Deaton (1999) examined education outcomes in South Africa using data collected in 1993, when government funding for schools was highly centralized and blacks (people of African descent) had virtually no political representation of any kind. The authors argue that blacks did not control the funds provided to their children’s schools and that tight migration controls limited their ability to migrate to areas with better schools. They show that pupil-teacher ratios varied widely across black schools, and argue that this variation, combined with migration barriers and black South Africans’ lack of control over their schools, generates a kind of natural experiment.

      Case and Deaton’s estimates indicate that raising school resources (as measured by student-teacher ratios) increases years of completed schooling and enrollment rates for blacks but not for whites. Since blacks had much larger class sizes than whites, this is consistent with the idea that there are diminishing returns to reductions in class size. They estimate large effects from reducing class size at black schools: decreasing the student teacher ratio from 40 to 20 (the approximate means in black and white schools, respectively) increases grade attainment by 1.5 to 2.5 years.

      Several issues raise concerns about the interpretation of these results. A key point is that, even if blacks could not influence class size in their children’s schools, someone, presumably some government officials, made decisions that influenced class sizes in South Africa’s black schools. If these decisions were influenced by education outcomes in those schools, or were merely correlated with such outcomes for some reason other than the causal impact of class size, they could yield biased estimates of the impact of class size (and, more generally, school resources) on those outcomes. This is the problem of endogenous program placement discussed in Section III. Another issue is that the children tested were not a random sample of household members, and data on student-teacher ratios from the Ministry of Education are not highly correlated—an R2 coefficient of 0.15—with the authors’ community data for that variable." Glewwe and Kremer, Pgs 22-23.

    • "The Case and Deaton (1999) analysis of South Africa, discussed above, also examined test scores. They found that decreasing the student-teacher ratio from 40 to 20 raises students’ reading test scores (conditional on years of school attendance) by an amount equivalent to the impact of two additional years of schooling. In contrast, there was no significant impact on mathematics scores." Glewwe and Kremer 2005, Pg 35.

23. 23.

    "Banerjee and others (2000) used a randomized evaluation to examine the impact of a program in India that attempted to raise school quality by hiring additional teachers, especially female teachers. An Indian NGO, Seva Mandir, runs nonformal schools that teach basic numeracy and literacy skills to children who do not attend formal schools and, in the medium term, attempts to “mainstream” these children into the regular school system. These schools are plagued by high teacher and child absenteeism, so the NGO decided to evaluate the impact of hiring a second teacher (where possible, a woman) in the hope of increasing the number of days the school was open, increasing student attendance, improving performance through individualized attention to students, and making school more attractive to girls. The program reduced the number of days a school was closed (one-teacher schools were closed 44 percent of the time, whereas two-teacher schools were closed 39 percent of the time), and girls’ attendance increased by 50 percent. However, the program had no significant effect on the attendance of boys." Banerjee 2000, Pg 28.

24. 24.

    "Glewwe, Ilias and Kremer (2004) do this, examining a randomized evaluation of the impact of a teacher incentives program in Kenya on both teacher behavior and test scores. They consider a model in which teachers can invest both in efforts to promote long-run learning and in short-run manipulation of test scores. Data were collected on many types of teacher effort—attendance, homework assignment, pedagogical techniques, and holding extra exam-preparation sessions—and on scores after the end of the program.

    The teacher incentive program in Kenya offered teachers prizes based on their schools' average scores on district-wide exams. The program penalized teachers for dropouts by assigning low scores to students who did not take the exam. During the two years the program was in place, student scores increased significantly in treatment schools (0.14 standard deviations above the control group). However, analysis of the Kenyan data suggests that this improvement did not necessarily occur through the channels intended. Teacher attendance and student dropout and repetition rates did not improve, and no changes were found in either homework assignment or pedagogy.

    Instead, teachers were more likely to conduct test-preparation sessions outside of normal class hours. Data from the year after the program ended show no lasting test score gains, which suggests that the teachers’ effort was concentrated in improving short-run outcomes, rather than stimulating long-run learning. The test-score effect was strongest for subject tests on geography, history, and Christian religion, arguably the subjects involving the most memorization. Also consistent with this hypothesis, the program had no impact on dropout rates, but exam participation rose (presumably because teachers wanted to avoid penalties for no-shows at exams)." Glewwe and Kremer 2005, Pg 43-44.

25. 25.

    "Finally, Banerjee and others (2004) recently conducted a randomized evaluation of a computer-assisted learning program in India and found much more positive results than those from the computer-assisted learning program in Israel (Angrist and Lavy, 2002). The idea of using computers in schools seems particularly promising in areas where both the number of qualified teachers and the quality of employed teachers is notoriously poor. The Indian CAL program took advantage of a donation by the state government of four computers to each municipal primary school in Vadodara and gave each child in the fourth standard 2 hours of shared computer time to play educational games that reinforced mathematical concepts (ranging from standard 1 to standard 3 levels). The program was found to be quite effective, with average mathematics score increases of 0.36 standard deviations in the first year and 0.51 standard deviations in the second year. The program was equally effective across student ability levels." Glewwe and Kremer 2005, Pg 39. See Banerjee 2004, Pgs 11-12 for details of the study.

26. 26.

    "This paper presents the evaluation of the program Computers for Education. The program aims to integrate computers, donated by the private sector, into the teaching of language in public schools. The authors conduct a two-year randomized evaluation of the program using a sample of 97 schools and 5,201 children. Overall, the program seems to have had little effect on students' test scores and other outcomes. These results are consistent across grade levels, subjects, and gender. The main reason for these results seems to be the failure to incorporate the computers into the educational process. Although the program increased the number of computers in the treatment schools and provided training to the teachers on how to use the computers in their classrooms, surveys of both teachers and students suggest that teachers did not incorporate the computers into their curriculum." Barrera-Osorio and Linden, abstract.

27. 27.

    "Vermeersch and Kremer (2004) conducted a randomized evaluation of the impact of school meals on participation in Kenyan preschools, and found that school participation was 30 percent greater in the 25 Kenyan preschools where a free breakfast was introduced than in the 25 comparison schools. There was some evidence the provision of meals cut into instruction time. In schools where the teacher was relatively well trained prior to the program, the meals program led to higher test scores (0.4 of a standard deviation) on academic tests. There were no effects on tests of general cognitive skills, implying the school meals program did not improve children’s nutritional status and that the academic test score increases were likely due to more time spent in school." Glewwe and Kremer 2005, Pg 25.

28. 28.

    "Kremer, Miguel, and Thornton (2004) conducted a randomized evaluation of the Girl’s Scholarship Program (GSP), introduced in rural Kenya in late 2001 to enhance girls’ education. Out of a set of 128 schools, half were randomly chosen as schools eligible for the program. The program consisted of a merit-based scholarship awarded to girls in two districts of Western Kenya who scored in the top 15 percent on tests administered by the Kenyan government. One portion of the scholarship was paid directly to the school for school fees, the other portion to the family for school supplies and uniforms. Girls eligible for the scholarship had significantly higher school attendance rates (as well as significantly higher test scores, average gains of 0.12-0.19 standard deviations). Schools offering the scholarship also had significantly higher teacher attendance after the program was introduced, and there is evidence of positive program externalities on boys (who were ineligible for the awards) as well as on girls with low pre-test scores (who were unlikely to win awards)." Glewwe and Kremer 2005, Pg 28.

29. 29.

    "A remedial education program in urban India, focused on improving the learning environment in public schools, appears to have increased test scores at a low cost. Banerjee and others (2004) conducted a randomized evaluation of a 2-year remedial education program in Mumbai and Vadodara, India. The remedial education program is run by a collaboration between a local NGO and the Indian government, and hires (at a yearly cost of only US$5 per child) young women from the community to teach basic literacy and numeracy skills to children who reach grade 3 or 4 without mastery of some basic competencies. On average, the program increased test scores by 0.14 standard deviations in the first year and 0.28 in the second year. The gains were largest for children at the bottom of the distribution, which is unusual for educational programs. Results were similar in both grade levels and in two different cities. The authors note that this program would be several times more cost-effective than hiring new teachers." Glewwe and Kremer 2005, Pg 38.

30. 30.

    Miguel and Kremer 2004.

31. 31.

    Kapur 2008, Pg 6. The figure given is the "gross enrollment ratio," which is the "total enrollment at a given educational level, regardless of age, divided by the population of the age group that typically corresponds to that level of education. The specification of age groups varies by country."

32. 32.

    Kapur and Crowley 2008, abstract.

33. 33.

    Behrman 2002, abstract.

34. 34.

    "Our measure of the dependent variable in relation (1) was generated by administering (in the regional language) to every person in our sample more than 10 years old, and with at least four years of schooling, tests of literacy and numeracy specially designed by the Educational Testing Service." Behrman 2002, Pg 8.

35. 35.

    Behrman 2002, Pg 6 Framework.

36. 36.

    Behrman 2002, Pg 34.

37. 37.

    Behrman 2002, Pg 35.

38. 38.
    • A causal relationship between A and B, and between B and C, does not imply a causal relationship between A and C. For example, cognitive skills may reflect both advantages in quality of schooling and other advantages in nutrition, motivation, intelligence, etc. Their relationship to earnings could be driven solely by the latter. In this case more schooling would lead to higher cognitive skills (independent of other factors) and cognitive skills would be associated with higher earnings (independent of the factors controlled for in the study), but this would still not imply that quality of schooling directly impacted earnings.
    • Unlike an "experiment" or "natural experiment," this paper relies on statistical adjustments to "control for" factors that may influence earnings/cognitive skills aside from quality of schooling. Such statistical adjustments rely on many assumptions about the measures of these factors and the nature of their relationships.

39. 39.
    "Economists often examine the relationship between years of schooling and income, yet more can be learned from examining the direct relationship between income and cognitive skills. First, positive correlation of cognitive skills with earnings, after conditioning on years of schooling, degrees obtained, and measures of innate ability, casts doubt on other interpretations of the correlation between income and education, such as claims that such correlation reflects only "sheepskin" effects, individuals' innate ability, or learned acquiescent behavior (Samuel Bowles and Herbert Gintis, 1976). (Sheepskin effects are increases in income solely due to possession of a diploma or other certificate, as distinct from any effect of skills acquired from the education that the diploma or certificate represents.)
    …
    Despite these potential benefits, there has been little research in both developed and developing countries on the relationship between cognitive skills and income. The main obstacle is the paucity of data sets that include both the incomes and the cognitive skills of adults, although the situation has improved in recent years. This section examines the evidence from developing countries on the impact of cognitive skills on incomes, both wage income and income from self-employment activities." Glewwe 2002, Pg 466.

40. 40.

    Glewwe 2002, Pgs 468-469.

41. 41.

    "The first is by Dean Jolliffe (1998), who estimates the impact of mathematics and reading skills on the agricultural, nonagricultural, and total income of 1388 Ghanaian households. He finds that cognitive skills raise nonfarm income and total income, but not farm income. This suggests low returns to numeracy and literacy in agricultural activities in Ghana, which induces households with relatively high skills to move out of farming and into non-farm activities.
    …
    Jolliffe's paper is quite innovative; perhaps the main criticism is what it did not examine. It never used the Raven's test data to see whether they have any explanatory power beyond that provided by cognitive skills and years of schooling. Also, it does not investigate whether schooling becomes insignificant when skills variables are added. Finally, it does not examine whether agricultural productivity per hour of work, rather than total income, was affected by cognitive skills, since the absence of an impact on total agricultural income may reflect decreased time spent in that activity. (Similarly, part of the positive impact of skills on non-farm income may reflect more hours in that activity.)
    …
    The only other study of the impact of cognitive skills on self-employment income is that of Wim Vijverberg (1999), who examined the same data used by Glewwe and Jolliffe. The author examined 1074 household enterprises in Ghana. Unlike the studies on wages and on total household income, he finds only weak evidence that schooling, measured either by years of school attendance or by cognitive skills, affects income from such enterprises. In fact, the impact of cognitive skills is often weaker than that of years of education.Yet there is one point of agreement with the wage studies: innate ability as measured by the Raven's test has no signifi- cant impact on income from nonfarm self-employment activities. Vijverberg con- cludes that impact of education on non- farm self-employment income is complex and probably varies by the type of business.
    Vijverberg's analysis has no serious flaws, but unlike Jolliffe he does little to account for sample selection bias." Glewwe (2002), Pgs 470-471.
    

42. 42.

    "A large body of literature investigates the impact of schooling infrastructure on schooling, as well as the returns to education in developing countries [see George Psacharopoulos (1994) and John Strauss and Duncan Thomas (1995) for surveys]. Estimated returns to education are, in general, larger in developing countries than in industrialized countries. However, most of the existing studies are based on simple correlations between years of education and wages. Family and community background are important determinants of both schooling and labor market outcomes in developing countries, and the bias inestimates that treat an individual’s education level as exogenous could be important." Duflo 2001, Pg 795.