Population Services International promotes and distributes materials, particularly condoms and bednets, to save lives and improve health throughout the developing world. Its largest program areas focus on preventing HIV/AIDS and reducing unwanted pregnancies (primarily through condom promotion/distribution) and reducing the burden of malaria (primarily by promotion/distribution of insecticide-treated nets (ITNs)).

When used appropriately, these materials can be very effective, at less than $1000 per infant death averted (nets) / HIV infection averted (condoms). We are not highly confident that this is happening consistently, but PSI is more transparent and results-driven than comparable charities and is likely having significant impact.

Note: many of the notes in this report reference our work on Population Services International from our 2007-2008 report.

What do they do?

Activities

PSI promotes and distributes materials, particularly condoms and bednets, to save lives and improve health in the developing world. Its largest program areas focus on preventing HIV/AIDS and reducing unwanted pregnancies (primarily through condom promotion/distribution) and reducing the burden of malaria (primarily by promotion/distribution of insecticide-treated nets (ITNs)). See our 2007-2008 discussion of PSI's activities, which gives funding allocation by product. A general update on the more recent funding allocation is given under 2009 funding by program.

PSI has historically focused on selling products for below-market prices, seen in our 2007-2008 discussion of this topic, but also distributes some products for free. According to its 2007 Annual Sales and Health Impact Report, most condoms distributed were sold, and all ITNs were sold.1 In May 2009, the Director of Research and Metrics, Kim Longfield, told us that free ITN distribution now accounts for a significant part of PSI's impact.2

Monitoring and evaluation

PSI expresses a commitment to relatively systematic monitoring and evaluation of its activities - particularly "MAP surveys," which aim to evaluate the availability (and appropriate marketing) of PSI's products in a region,3 and "TRaC surveys," which aim to evaluate relevant behavior change over time4 (for example, use of condoms and insecticide-treated nets - see the following section).

Results from TRaC reports are available online on PSI's website.5 Technical papers describing the methodology for data collection for specific TRaC surveys are also available on PSI's website.6

The surveys published online are numerous but not complete (details in the following section). Kim Longfield stated to us that many reports are sent to the U.S. office behind schedule (or not at all), which is the reason for the missing reports.7 It appears that no single overview or summary of these surveys are available, although we give a summary of publicly available data in the following section.

Does it work?

Both condoms and ITNs (which together comprise the bulk of PSI's expenses, as noted above) are proven to improve health (reducing the risk of HIV infection and malaria, respectively) when used correctly and consistently. For more, see our full report on the promotion and distribution of condoms and ITNs.

The mere fact that PSI sells these products does not by itself imply that its customers are using them correctly and consistently. PSI conducts many surveys attempting to assess whether behavioral patterns are changing (as discussed in the previous section).

We reviewed the results of these surveys for all projects that have published data on PSI's website, looking at changes in condom use (for PSI's condom program)8 and changes in (a) ITN ownership and (b) ITN use by pregnant women and children under 5 (for the ITN program)9. In brief:10

• Condoms: Those surveyed in approximately 40 PSI projects showed a small (1-4%) average increase in condom use between the baseline survey (usually given in 2004) and followup survey (usually given in 2007). Condom use across African countries as a whole seems to follow a similar pattern.11

• ITNs: Surveys from 6 PSI projects showed a large increase in ITN ownership (17% average arithmetic increase) and use (11% among pregnant mothers; 16% among children under 5) in published surveys. We were unable to find independent survey data to determine whether these changes correspond to broader trends within Africa.

  We are aware of at least one country where there was significant ITN-distribution activity outside of PSI: Kenya. We discuss Kenya's ITN distribution program here; studies of the program credit PSI with a major role, but not with sole responsibility for the encouraging results. We find it slightly concerning that PSI's discussions of trends within Kenya do not mention the context of the large-scale government-led program that was taking place at the time.

Three other notes about the question of PSI's impact:

• PSI appears to be a dominant supplier of both condoms and ITNs within Africa.12 Thus, country- and continent-wide trends might be legitimately attributable to PSI's activities.

• Survey data implies that condom use among PSI's targeted groups is generally relatively high,13 although use of ITNs varies extremely widely.14 In the former case, there is some argument that the simple fact of high condom utilization rates implies that the condoms sold by PSI are likely being used.

• It can be argued that because PSI often sells products rather than giving them away (see discussion above), its products are unlikely to be wasted. However, because PSI sells at significantly below-market prices (see the discussion in our 2007-2008 review), it is possible that people buy them primarily to re-sell them, and that they do not ultimately end up benefiting low-income people.

Possible negative or offsetting impact

PSI sells products at significantly below-market prices (see the discussion in our 2007-2008 review), and may thus be effectively replacing or outcompeting private provision of these products. We have little sense of the size of this potential problem. PSI did send us a paper proposing a method for assessing the value-added while considering the presence of other suppliers,15 but we have not seen its framework applied in PSI's actual monitoring and evaluation.

What do you get for your dollar?

In our 2007-2008 report, we roughly estimated that PSI's condom program prevents a case of HIV/AIDS for every every 500-1,000 it spends, and that its ITN program prevents a case of malaria for every $600-2,400 it spends.16 At that time, we did not have access to the details of PSI's own calculations; we now have the details of its calculations for both condoms17 and ITNs.18 We find its approach to be largely similar to the approach we took, but with more detail and more informational sources than we used. We therefore defer to PSI's own cost-effectiveness estimates, with the following caveats:

• PSI's condom estimate assumes a 10% wastage rate for purchased condoms19 and a 35% wastage rate for free condoms.20 No source is given for these estimates, and it appears that PSI's monitoring and evaluation does not include methods for determining actual wastage rates. We believe that the question of whether the condoms sold by PSI reach low-income people who use them consistently and appropriately is a major question.

• Similarly, PSI's estimate for ITNs includes the assumption that "each LLIN [a form of ITN] sold will protect on average 1.42 people, of which 0.55 of those will be aged 0-4 years," citing unspecified "studies."21 Given the high variation in ITN usage rates (described above) and the relatively small amount of data on them, we find the appropriateness of this assumption be particularly questionable.

It is normal for cost-effectiveness estimates to include many rough estimates and assumptions, but we have highlighted those that pertain directly to the efficacy of PSI's programs - i.e., we have highlighted questions about whether PSI's products are used as intended.

PSI's most recent report on cost-effectiveness gives overall estimates by product including:

• Condoms: $28.08 per disability-adjusted life-year22 (DALY, a metric discussed here), equivalent to ~$550 per HIV infection averted.23 This figure is noticeably lower than, though still within an order of magnitude of, an estimate of $52-112 per DALY averted (implying $1,040-$2,240 per HIV infection averted) by an independent source (details here).

• ITNs: $12.38-$18.82 per DALY averted,24 equivalent to ~$380-580 per infant death averted.25 This figure is relatively consistent with other estimates of the cost-effectiveness of ITNs (more at our analysis of ITN cost-effectiveness).

Room for more funds?

We have not seen a formal analysis of PSI's plans for future funds or of its perceived "funding gap." Kim Longfield stated to us that unrestricted funds are granted to regional programs based on a competitive application process.26 Based on our financial analysis,27 it appears that PSI has been expanding its focus on ITNs, and we believe it would be reasonable to expect further expansion to result in more ITN coverage.

Financials/other

Unless otherwise noted, all data comes from PSI's IRS form 990s for 2002-2007, available for download through the National Center for Charitable Statistics.28

Revenue and expense growth (about this metric): PSI's revenues and expenses have grown consistently over the past 6 years.

Assets-to-expenses ratio (about this metric): PSI maintains a low assets:expenses ratio. This may be because PSI receives a large portion of its support from governments, which may provide less volatile support, so that PSI does not need as large a reserve cushion as other charities.

Expenses by program area (about this metric): PSI spends the majority of its funds on its rural health clinics, the focus of our review.29

Expenses by IRS-reported category (about this metric): PSI maintains a reasonable "overhead ratio," spending approximately 90% of its budget on program expenses.

Unanswered questions

• Room for more funds. Does PSI have "funding gap" analysis available?
• Possible negative or offsetting impact. How does PSI determine whether it is adding to the supply of materials or replacing/outcompeting private sector provision of these materials?

Sources

• GiveWell/PSI. 2009. "Summary of PSI TRaC Performance Data. "Available online here. Data compiled by GiveWell and sourced from PSI's website at http://www.psi.org/resources/research-metrics/studies, accessed 1/8/10.
• Givewell. 2008. "PSI 2007-08 Review." Available at www.givewell.net/node/241, accessed 6/30/09.
• Longfield, Kim. PSI Director of Research. Phone conversation, 5/26/09
• PSI. 2006. "Cost-Effectiveness Report." Available online, accessed 6/31/09.
• PSI. 2009. "Funding by program." PSI internal document, available online here.
• PSI. 2009. "Quarterly Progress Report, 2007-2009." PSI internal document, available online here.
• PSI. 2009. "Scorecard. April 2009." PSI internal document, available online here.
• PSI. "DELTA Marketing Lesson." PSI internal document, available online here.
• PSI. "Condom DALY Brief." PSI internal document, available online here.
• PSI. "Condom DALY Coefficient Worksheet." PSI internal document, available online here.
• PSI. "Condom DALY Model Description." PSI internal document, available online here.
• PSI. "LLIN DALY Brief." PSI internal document, available online here.
• PSI. "Total Market Approach." PSI has asked that we keep this document confidential.
• PSI. Organization website. Online at http://www.psi.org/, accessed 1/8/10.

1. 1.

   Data collected from 2007 Annual Sales and Health Impact Report pages 24, 46, 62, 77-78, 95-96, 120. The number of "free condoms" is about 5% of the total of "free condoms" and "condoms." The only categories that appear to refer to free ITNs ("LLIN free") are blank. (LLIN acronym defined at http://acronyms.thefreedictionary.com/LLIN accessed 6/25/09)

2. 2.

   Phone conversation with Kim Longfield, PSI's Director of Research, 5/26/09.

3. 3.

   PSI Organization Website. See http://www.psi.org/resources/research-metrics/studies/map, accessed 1/8/10.

4. 4.

   PSI Organization Website. See http://www.psi.org/resources/research-metrics/studies/trac, accessed 1/8/10.

5. 5.

   PSI Organization Website. See http://www.psi.org/resources/research-metrics/studies/trac, accessed 1/8/10.

6. 6.

   http://www.psi.org/research/cat_socialresearch_smr.asp, accessed 6/30/09

7. 7.

   Phone conversation with Kim Longfield, PSI Director of Research, 5/26/09.

8. 8.

   Definition of 'condom use':
   "The 'C indicator' refers to condom use at last sex with a non-marital noncohabiting partner in the past 12 months. For sex workers, the 'C indicator' refers to the percentage of sex workers who used a condom at their last sex with a client, of all those who report having sex for money or material goods. For men who have sex with men, the 'C indicator' refers to the percentage of men who used a condom at last anal sex with a male partner, of those who have had anal sex with a male partner in the past 12 months." PSi organization website. See http://web.archive.org/web/20070901145420/http://www.psi.org/research/st..., accessed 1/12/10.

9. 9.

   Definition of ITN use: "'Pregnant Women' contain two indicators: usage of net and usage of ITN, which refer to the percentage of pregnant women who slept under a mosquito net or an ITN the previous night, respectively. 'Children Under Five' contains two indicators: usage of net and usage of ITN, which refer to the percentage of children under five who slept under a mosquito net or an ITN the previous night, respectively. Some malaria surveys contained only information at the household level; others contained information at the individual level, or could be converted from household to individual data." PSI organization website. See http://web.archive.org/web/20070901145420/http://www.psi.org/research/st..., accessed 1/12/10.

10. 10.

    GiveWell/PSI "Summary of PSI TRaC Performance Data."

11. 11.

    See http://www.givewell.net/node/241#CondomUseChanges

12. 12.

    See the discussion from our 2007-2008 review. In addition, "Condom DALY Brief" states that "Market share of PSI condoms is about 20%", Pg 1.

13. 13.

    On 6/26/09, we compiled all of PSI's publicly availablesurvey results, available at http://www.psi.org/resources/research-metrics/studies (accessed 1/8/10), focusing on the "most recent round" figure for the "Condom use (15-49 years)" question, which asks whether respondents used a condom at their last sexual act (see definitions at http://www.psi.org/research/studiesStartpage.htm). 121 total figures were available; 104 of these indicated over 50% condom usage at last sexual act, while 58 indicated over 80%. These numbers are broadly consistent with (although measured very differently from) the assumptions about condom use that PSI uses in its cost-effectiveness estimates for condoms: "Condom usage is estimated to be 10% with regular partner, 40% with casual and 60% with commercial partner" ("Condom DALY Brief").

    There is some risk that survey data overstates condom use, though we feel the risk is smaller with a factual question about recent activity (the question we examined) than it would be with more general questions about condom use.

    Our compilation of this data is available in the Excel file, GiveWell/PSI "Summary of PSI TRaC Survey Data" 2009.

14. 14.

    On 6/25/09, we scanned all of PSI's publicly available survey results, available at http://www.psi.org/resources/research-metrics/studies (accessed 1/8/10), focusing on the "most recent round" figure for the "Children under 5" question, which asks whether children under 5 in respondents' households slept under a net the previous night. Only 12 total figures were available, ranging from 4.4%/4.6% in one case to 55.8%/40.3% in another (and 93%/19.9% in another). In no case were both figures over 45%. (Note that the first figure refers to sleeping under any kind of mosquito net, while the second is specific to ITNs - see definitions at http://web.archive.org/web/20070901145420/http://www.psi.org/research/st..., accessed 1/12/10.)

    There is some risk that survey data overstates condom use, though we feel the risk is smaller with a factual question about recent activity (the question we examined) than it would be with more general questions about ITN use.

    Our compilation of this data is available in the Excel file, GiveWell/PSI "Summary of PSI TRaC Survey Data" 2009.

15. 15.

    PSI "Total Market Approach."

16. 16.

    See our 2007-2008 cost-effectiveness analysis.

17. 17.

    See PSI "Condom DALY Brief", "Condom DALY model description" and "Condom DALY Coefficient Worksheet."

18. 18.

    PSI "LLIN DALY Brief."

19. 19.

    PSI "Condom DALY Brief," Pg 2.

20. 20.

    "The approach applies to free condom as well except that we assume an additional 25% wastage rate." PSI "Condom DALY brief," Pg 4.

21. 21.

    PSI "LLIN DALY Brief," Pg 1.

22. 22.

    PSI "Cost-Effectiveness Report" 2006, Pg 8.

23. 23.

    "PSI assumes all HIV infected individuals will die after 10 years of HIV infection and two years of AIDS. The DALYs averted for averting one new infection is 19.3 DALYs for the lower three risk groups, 19.5 DALYs for the very high risk group, and 19.7 DALYs for the highest risk group." "$28.80 per DALY implies a cost per infection averted of $542 (at 19.3 DALYs per infection averted) to $553 (at 19.7 DALYs per infection averted)." PSI "Condom DALY brief," Pg 2.

24. 24.

    PSI "Cost-Effectiveness Report" 2006, Pg 8. Note that "Olysets" and "LLINs" are both types of ITNs.

25. 25.

    "The value for the life of a child is approximately 31 YLL." PSI "LLIN DALY brief," Pg 1. See our discussion of DALYs for clarification on the YLL term; 1 YLL is equivalent to 1 DALY.

26. 26.

    Phone conversation with Kim Longfield, PSI Director of Research, 5/26/09.

27. 27.
    • Organization-level financials imply room/plans for expansion.

    • Expenses by activity from 2007-2008 report.
    • Update on activities shows a growing malaria program.

28. 28.

    http://nccsdataweb.urban.org/PubApps/showVals.php?ft=bmf&ein=560942853, accessed 12/25/09.

29. 29.

    Data comes from PSI "Cost-effectiveness report" 2007, Pg 5. Available online at http://www.psi.org/sites/default/files/publication_files/2007_Cost_Effec..., accessed 12/25/09.