This page logs mistakes we've made, strategies we should have planned and executed differently, and lessons we've learned.

Because we are a startup organization working in areas we have little experience with, it is particularly important that we constantly recognize and learn from our shortcomings. We make this log public so as to be up front with any potential supporters about ways in which we need to improve.

A full review of our first year – both accomplishments and shortcomings - is available here.

Please use our survey, or just contact us, with other items that should be listed here.

Major issues

12/2007: overaggressive and inappropriate marketing

How we fell short: As part of an effort to gain publicity, GiveWell's staff (Holden and Elie) posted comments on many blogs that did not give adequate disclosure of our identities (though we did use our real first names); in a smaller number of cases, we posted comments and sent emails that deliberately concealed our identities. Our actions were wrong and rightly damaged GiveWell's reputation. More detail is available via the page for the board meeting that we held in response.

Given the nature of our work, it is essential that we hold ourselves to the highest standards of transparency in everything we do. Our poor judgment caused many people who had not previously encountered GiveWell to become extremely hostile to it.

Steps we have taken to improve: We issued a full public disclosure and apology, and directly notified all existing GiveWell donors of the incident. We held a Board meeting and handed out penalties that were publicly disclosed, along with the audio of the meeting. We increased the Board's degree of oversight over staff, particularly with regard to public communications.

5/2008-12/22/2008: failed to track website traffic/statistics

How we feel short: We discovered on 12/22/2008 that we had not been tracking any visits to our site since May of that year. At the time, we were not emphasizing website traffic as a metric, and so had not been checking up on it regularly. Failing to do so meant that we lost valuable data that could have been used to measure our progress over time. Over a year later, our attempts to review our progress have been hampered by the fact that this data is missing.

Steps we have taken to improve: we reinstituted tracking immediately upon discovering our error, and have since checked up on the tracking more regularly. Going forward, we plan to review key website metrics on a quarterly basis.

6/2008-9/2008: business plan over-focused on marketing, under-focused on research

How we fell short: after completing our first year of research, we agreed that the focus of our second year should be on increasing the "money moved" by our research. (Details here.) After several months focusing on this goal, we felt that we were spending insufficient time on research - the core of our mission - and weren't getting enough return out of the time we were spending on marketing. (Details here.) We now believe that we over-focused on marketing, at a stage in our development where doing so was premature, and as a result did not make as much progress during these months as we should have.

Steps we have taken to improve: we created a new business plan that shifted our focus primarily to research; this business plan, along with companion documents that lay out the reasons for our change of direction, is available here.

6/2007: poorly constructed "causes" led to suboptimal grant allocation

How we fell short: For our first year of research, we grouped charities into causes ("Saving lives," "Global poverty," etc. - see the purple bar at the top of this page) based on the idea that charities within one cause could be decided on by rough but consistent metrics: for example, we had planned to decide Cause 1 (saving lives in Africa) largely on the basis of estimating the “cost per life saved” for each applicant. The extremely disparate nature of different charities’ activities meant that there were major limits to this type of analysis (we had anticipated some limits, but we encountered more).

Because of our commitment to make one grant per cause and our overly rigid and narrow definitions of "causes," we feel that we allocated our grant money suboptimally. For example, all Board members agreed that we had high confidence in two of our Cause 1 (saving lives) applicants, but very low confidence in all of our Cause 2 (global poverty) applicants. Yet we had to give equal size grants to the top applicant in each cause (and give nothing to the 2nd-place applicant in Cause 1).

Steps we have taken to improve: We have shifted our approach to "causes" so that they are defined more broadly. This gives us more flexibility to grant the organizations that appeal to us most. We now explore broad sets of charities that intersect in terms of the people they serve and the research needed to understand them, rather than narrower causes based on the goal of an “apples to apples” comparison using consistent metrics. For example, our recent research report addresses the broad area of international aid.

6/2007-11/2007: set overly ambitious deadlines for completing our research

How we fell short: In July 2007, when we mailed out grant applications to charities, we told them that we would award grants by December 2007. We did not finish all necessary research by December 2007, and thus only paid out 3 of our planned 5 grants by the agreed upon date. We paid the last two grants in early March of 2008.

What we are doing to improve: Setting deadlines was particularly difficult for our first year, as we had no previous experience with grantmaking. With the first year behind us, we have a better sense of how time-consuming research is, and we have sinceset more conservative expectations. Our 2008-2009 report on international aid was delivered by the deadline we had set.

6/2007-5/2008: research process relied excessively on open-ended grant applications

How we fell short: In our first year, we focused our time and effort overwhelmingly on getting information from applicants, as opposed to from academic and other independent literature. Applicants found this process extremely time-intensive and burdensome, particularly given the size of the grants. We also found that much of the information we found essential in making informed decisions was not submitted through grant applications (rather, we found it through independent research).

Steps we have taken to improve: We have modified our research process. Our basic process now (for the latest implementation, see our research process for international aid is to (a) search independent research to identify particularly promising approaches; (b) use heuristics to identify charities that are both transparent (i.e., sharing substantial information) and promising; (c) contact charities with specific and targeted questions. We feel that this process leads both to better information and to a lower burden on charities.

Smaller issues

5/20/09-8/27/09 and 10/25/2009-11/11/2009: careless with updates to website; broke email update signup form

In 2009, we made multiple changes to our website with the help of a contractor, and we were not sufficiently careful about checking all changes to make sure they retained all functionality. As a result, website visitors lost the ability to sign up for email updates (from the front page - other forms worked) between 5/20/09 and 8/27/09 and between 10/25/2009 and 11/11/2009.

Steps we have taken to improve: we try to make all changes on a "mirror site" first (we had a mirror site at the time but were bypassing it for multiple small changes). When we make a change directly to the live site, we test relevant functionality immediately afterward.

5/2009-8/2009: excessive time spent on policies and procedures

In the process of our 2008 financial audit, we decided to draft a comprehensive set of internal policies, employee manual, etc.

Doing so turned out to be beyond our capacity as a small organization. We ultimately decided to maintain only the most essential policies, which are now posted online. All in all, the process ended up costing us significant time, and we struggled to file our return by the final deadline (although we did do so).

1/2009-3/2009: poor research strategy

While conducting research for our 2008-2009 international aid report, we found ourselves very impressed with the Carter Center, and put a large amount of time into understanding - and writing up - its programs that most appealed to us. However, we had neglected to first establish the question of how its funding was allocated between programs. We had a general sense that the programs that appealed to us were also the largest programs, but after further investigation we found ourselves continually unable to verify this. As of November 2009, we still had not gotten the basic information we needed to have confidence in the Carter Center.

If we had focused on the basics (on which programs does the Carter Center focus?) first, we would have saved the substantial time we put into its review.

Steps we have taken to improve: we have become more disciplined about the order in which we ask questions about a charity. First, we need to understand where its funds are going and what it does; only then do we decide which programs to investigate deeply.

1/2008-9/2008: paying insufficient attention to professional development and support

How we fell short: at our board meeting in January 2008, we agreed to explore options for professional development and mentoring, in light of the relative youth and inexperience of our staff. GiveWell staff put a lower priority on this than more time-sensitive goals, and while we explored a few options, we made little progress on it between January and September. At the September Board meeting, the Board criticized this lack of progress and reiterated the need for professional development and mentoring.

Steps we have taken to improve: we now have two highly regular mentoring relationships, and two more in a "trial phase." We have also stepped up Board oversight through a monthly conference call (attendance is optional but has generally been high) and more regular calls with Board Vice-President Lindy Miller. An update on professional development was presented at our July 2009 Board meeting.

6/2008: overly aggressive time estimates

How we fell short: The timeline presented in our year-2 plan (published 6/19/2008) called for us to finish a round of both marketing (raising GiveWell Pledges) and research (publishing our full report on developing-world aid) in December 2008. We had little to go on in forming this estimate, and largely chose this target date because of the significance of the holiday season in giving (rather than based on quantified time estimates).

Steps we have taken to improve: we have been recording our hours since January of 2008, and have gradually been improving the detail of our timesheets. We now have enough data on how our time is spent to make more detailed estimates. Our 2008-2009 international aid report was published on 7/1/2009, consistent with the goals we had set at the end of 2008.

2/2008-5/2008: premature hiring

How we fell short: In an attempt to increase our research capacity, we hired our strongest volunteer in January 2008, but terminated the relationship at the end of May. We have mutually agreed that, at this stage of our development, we can't provide the training and management necessary for someone of his skill set to add significant value.

Steps we have taken to improve: We paused recruiting with the intent of picking it back up when we had a more systemized process. We went through a series of hires in early 2009, but at that time we still did not have a systemized enough process, and relationships were terminated fairly quickly. In mid-2009, we again reassessed our situation and made two hires who lasted substantially longer. One worked for us from May through December of 2009, added significant value, and left in December 2009 for an opportunity in the consulting industry; another started work in early July 2009 and is still with us (and adding significant value) as of December 2009.

12/2007-present: website not sufficiently engaging, has generated too little substantive feedback

How we fell short: We believe that our current reviews, while thorough, are overly dense and difficult to engage with. Feedback from our supporters has often included this theme. In addition, we have received less little critical engagement with our analysis than we would like from those not directly involved in the project. We have received many emails offering general support or asking us to consider a particular charity, but the number of people who have critiqued the content of our reviews – through email, survey, discussion forum, or our blog – remains low. Our site is designed so that people can take quick action based on our recommendations, or dig into the details of our reasoning; we suspect that many people have opted for the former, and very few have opted for the latter.

Steps we have taken to improve: We devoted substantial time over the 8/2009-12/2009 period to revamping our website (for example, making summary information by topic easier to find) and to making frequent posts on our blog that present our research in more accessible ways. We have also actively sought out feedback from relevant experts.

The level of feedback has improved, but our website still does not present our research nearly as accessibly as it could.

12/2007-5/2008: research process should have incorporated more and earlier discussions with charities' staff members

How we fell short: We undervalued the discussions with staff during our first year; speaking directly with staff gives us the opportunity to get a clear picture of how an organization views itself, and therefore what sorts of information we should seek to get a picture of whether its approach works as intended. Applicants encouraged us to put more of our time into personal visits, so that we could “get to know” organizations rather than thrusting pre-defined questions on them.

Steps we have taken to improve: We now begin any charity evaluation (once the initial heuristics have been passed and we've learned what we can from the website) with discussions with staff.