This page logs mistakes we've made, strategies we should have planned and executed differently, and lessons we've learned.
Because we are a startup organization working in areas we have little experience with, it is particularly important that we constantly recognize and learn from our shortcomings. We make this log public so as to be up front with any potential supporters about ways in which we need to improve.
A full review of our first year – both accomplishments and shortcomings - is available here.
Please use our survey, or just contact us, with other items that should be listed here.
How we fell short: As part of an effort to gain publicity, GiveWell's staff (Holden and Elie) posted comments on many blogs that did not give adequate disclosure of our identities (though we did use our real first names); in a smaller number of cases, we posted comments and sent emails that deliberately concealed our identities. Our actions were wrong and rightly damaged GiveWell's reputation. More detail is available via the page for the board meeting that we held in response.
Given the nature of our work, it is essential that we hold ourselves to the highest standards of transparency in everything we do. Our poor judgment caused many people who had not previously encountered GiveWell to become extremely hostile to it.
What we are doing to improve: We issued a full public disclosure and apology, and directly notified all existing GiveWell donors of the incident. We held a Board meeting and handed out penalties that were publicly disclosed, along with the audio of the meeting. We increased the Board's degree of oversight over staff, particularly with regard to public communications.
How we fell short: after completing our first year of research, we agreed that the focus of our second year should be on increasing the "money moved" by our research. (Details here.) After several months focusing on this goal, we felt that we were spending insufficient time on research - the core of our mission - and weren't getting enough return out of the time we were spending on marketing. (Details here.) We now believe that we over-focused on marketing, at a stage in our development where doing so was premature, and as a result did not make as much progress during these months as we could have.
What we are doing to improve: we have created a new business plan that shifts our focus primarily to research; this business plan, along with companion documents that lay out the reasons for our change of direction, is available here.
How we fell short: For our first year of research, we grouped charities into causes ("Saving lives," "Global poverty," etc. - see the purple bar at the top of this page) based on the idea that charities within one cause could be decided on by rough but consistent metrics: for example, we had planned to decide Cause 1 (saving lives in Africa) largely on the basis of estimating the “cost per life saved” for each applicant. The extremely disparate nature of different charities’ activities meant that there were major limits to this type of analysis (we had anticipated some limits, but we encountered more).
Because of our commitment to make one grant per cause and our overly rigid and narrow definitions of "causes," we feel that we allocated our grant money suboptimally. For example, all Board members agreed that we had high confidence in two of our Cause 1 (saving lives) applicants, but very low confidence in all of our Cause 2 (global poverty) applicants. Yet we had to give equal size grants to the top applicant in each cause (and give nothing to the 2nd-place applicant in Cause 1).
What we are doing to improve: We plan to research fewer, broader causes in the future, giving ourselves more flexibility to grant the organizations that appeal to us most. We plan to explore broad sets of charities that intersect in terms of the people they serve and the research needed to understand them, rather than narrower causes based on the goal of an “apples to apples” comparison using consistent metrics.
How we fell short: In July 2007, when we mailed out grant applications to charities, we told them that we would award grants by December 2007. We did not finish all necessary research by December 2007, and thus only paid out 3 of our planned 5 grants by the agreed upon date. We paid the last two grants in early March of 2008.
What we are doing to improve: Setting deadlines was particularly difficult for our first year, as we had no previous experience with grantmaking. With the first year behind us, we have a better sense of how time-consuming research is, and in the future we will set more conservative expectations.
How we fell short: In our first year, we focused our time and effort overwhelmingly on getting information from applicants, as opposed to from academic and other independent literature. Applicants found this process extremely time-intensive and burdensome, particularly given the size of the grants. We also found that much of the information we found essential in making informed decisions was not submitted through grant applications (rather, we found it through independent research).
What we are doing to improve: We are modifying our research process. In the future, we will start by searching independent research to identify particularly promising approaches; we will send out applications (for our grants and recommendations) later in the process, and while we will still invite charities to make their case in whatever way they deem appropriate, we will treat the application primarily as a way of getting specific information about specific organizations rather than as the starting point of our research. We believe that doing so will improve our ability to collect information, and lower the burden on applicants.
How we fell short: at our board meeting in January 2008, we agreed to explore options for professional development and mentoring, in light of the relative youth and inexperience of our staff. GiveWell staff put a lower priority on this than more time-sensitive goals, and while we explored a few options, we made little progress on it between January and September. At the September Board meeting, the Board criticized this lack of progress and reiterated the need for professional development and mentoring.
What we are doing to improve: we are now sending biweekly reports to two pending mentors (along with our Board), and are slowly seeking more. Our full plan for professional development and mentoring was presented at our November 2008 Board meeting.
How we fell short: The timeline presented in our year-2 plan (published 6/19/2008) called for us to finish a round of both marketing (raising GiveWell Pledges) and research (publishing our full report on developing-world aid) in December 2008. We had little to go on in forming this estimate, and largely chose this target date because of the significance of the holiday season in giving (rather than based on quantified time estimates).
What we are doing to improve: we have been recording our hours since January of 2008, and have gradually been improving the detail of our timesheets. We now have enough data on how our time is spent to make more detailed estimates, such as those in our more recent business plan.
How we fell short: In an attempt to increase our research capacity, we hired our strongest volunteer in January, but terminated the relationship at the end of May. We have mutually agreed that, at this stage of our development, we can't provide the training and management necessary for someone of his skill set to add significant value.
What we are doing to improve: We have determined that for the time being, capacity building is not a primary goal; the overhead and risk involved in finding and incorporating a new hire would interfere too heavily with higher priorities at this point. This year, we are focused on testing and demonstrating that demand for rigorous research into giving options exists. If and when we answer that question in the affirmative, the next major priority will be building capacity.
How we fell short: We believe that our current reviews, while thorough, are overly dense and difficult to engage with. Feedback from our supporters has often included this theme. In addition, we have received relatively little critical engagement with our analysis from those not directly involved in the project. Our discussion forum has gone practically unused, despite being linked from the top of every review page. We have received many emails offering general support or asking us to consider a particular charity, but the number of people who have critiqued the content of our reviews – through email, survey, discussion forum, or our blog – remains very low. Our site is designed so that people can take quick action based on our recommendations, or dig into the details of our reasoning; we suspect that many people have opted for the former, and very few have opted for the latter.
What we are doing to improve: We believe that we can do a much better job presenting the most relevant and interesting information up front rather than sticking to the highly systematic approach we’ve used to date (which tends to present information in a fixed, consistent order rather than in order of importance and interest). Our research summary, and the documents linked from the top of it, represent a preliminary attempt to do so. We believe that the more conversations we have with supporters, identifying the information that stands out to them, the better we will be able to design our reviews in a way that encourages engagement and interest. It is important to note that improving the website is not a major priority over the next year, as we plan to focus more on personal fundraising (see our plan for more).
How we fell short: We undervalued the role of the site visit during our first year; speaking directly with staff gives us the opportunity to get a clear picture of how an organization views itself, and therefore what sorts of information we should seek to get a picture of whether its approach works as intended. Applicants encouraged us to put more of our time into personal visits, so that we could “get to know” organizations rather than thrusting pre-defined questions on them.
What we are doing to improve: We have found site visits extremely useful in forming a picture of an organization and determining the most relevant questions for it. In the future, we will be modifying the structure of our research process, conducting site visits before we ask applicants to submit detailed organization-specific applications. We believe that doing so will lead to better-constructed applications that give us more information while imposing less burden on applicants.
