Joi Ito's Web

Joi Ito's conversation with the living web.

PhotologTue, Jun 2, 20:07 UTC

If you looked at how many people check books out of libraries these days, you would see failure. Circulation, an obvious measure of success for an institution established to lend books to people, is down. But if you only looked at that figure, you'd miss the fascinating transformation public libraries have undergone in recent years. They've taken advantage of grants to become makerspaces, classrooms, research labs for kids, and trusted public spaces in every way possible. Much of the successful funding encouraged creative librarians to experiment and scale when successful, iterating and sharing their learnings with others. If we had focused our funding to increase just the number of books people were borrowing, we would have missed the opportunity to fund and witness these positive changes.

I serve on the boards of the MacArthur Foundation and the Knight Foundation, which have made grants that helped transform our libraries. I've also worked over the years with dozens of philanthropists and investors--those who put money into ventures that promise environmental and public health benefits in addition to financial returns. All of us have struggled to measure the effectiveness of grants and investments that seek to benefit the community, the environment, and so forth. My own research interest in the begun to analyse the ways in which people are currently measuring impact and perhaps find methods to better measure the impact of these investments.

As we see in the library example, simple metrics often aren't enough when it comes to quantifying success. They typically are easier to measure, and they're not unimportant. When it comes to health, for example, iron levels might be important, but anemia isn't the only metric we care about. Being healthy is about being nourished and thus resilient so that when something does happen, we recover quickly.

Iron levels may be a proxy for this, but they aren't the proxy. Being happy is even more complicated; it involves health but also more abstract things such as feelings of purpose, belonging to a community, security, and many other things. Similarly, while I believe rigor and best practices are important and support the innovation and thinking going into these metrics when it comes to all types of philanthropy, I think we risk oversimplifying problems and thus having the false sense of clarity that quantitative metrics tend to create.

One of the reasons philanthropists sometimes fail to measure what really matters is that the global political economy primarily seeks what is efficient and scalable. Unfortunately, efficiency and scalability are not the same as a healthy system. In fact, many things that grow quickly and without constraints are far from healthy--consider cancer. Because of our belief in markets, we tend to accept that an economy has to be growing for society to be healthy--but this notion is misguided, particularly when it comes to things we consider social goods. If we examine a complex system like the environment, for instance, we can see that healthy rainforests don't grow in overall size but rather are extremely resilient, always changing and adapting.

There is more to assessing a complex system than looking at its growth, efficiency, and the handful of other qualities that can be quantified and thus measured.

As biologists know, healthy ecosystems are robust and resilient. They can tolerate reductions in certain species populations ... until they can't. Scholars in ecology and biology have tried to model the robustness and resilience of systems in an effort to understand how to build and maintain such systems. Scientists have tried to apply these models to non-biological systems like the internet and ask questions, such as "How many and which nodes can you remove from the internet before it stops functioning?" These models are different from the mathematics economists use. Instead of relying on aggregate numbers and formulae, they use network models of nodes and links to ponder dynamics among connections in the system, rather than stocks and flows of economies.

Maybe there is something to learn from biologists and ecologists--the people who study the complex and messy real world of nature--when philanthropists are thinking about how to save the planet. We know from ecology and biology, for instance, that monocultures and simple approaches tend to be weak and fragile. The strongest systems are highly diverse and iterate quickly. When the immune system goes to war against a pathogen, the body engages in an arms race of mutations, deploying a diversity of approaches and constant iteration, communication, and coordination. Scientists also are learning that the microbiome, brain, and immune system are more integrated and complex than we ever imagined; they actually understand and tackle the more complex diseases currently beyond our scientific abilities. This research is pushing biology and computational models to a whole new and exciting level.

Many diseases, just like all of the systems that philanthropy tries to address, are complex networks of connected problems that go beyond any one specific pathway or molecule. Obesity is often described as simply a matter of managing one's calories and consequently cast as a lack of willpower on the part of an overweight individual. But it is probably more accurately understood in the context of a global food system that is incentivized by financial markets to produce low cost, high-calorie, unhealthy, and addictive foods. Calorie counting as the primary way to lose weight has been a rule of thumb, but we are learning that healthy fats are fine while sugar calories cause insulin resistance, which often leads to diabetes and obesity. So solving the obesity problem is going to require much more than increasing or reducing any one single thing like calories.
It's our food system that is unhealthy, and one result is overweight individuals.

In such a complex world, what are we to do? We need respect for plurality and heterogeneity. It's not that we shouldn't measure things, but rather that we should measure different things, have different approaches and iterate and adapt. This is how nature builds resilient networks and systems. Because we as a society have an obsession with scale and other common measures of success, researchers and do-gooders have a natural tendency to want to use simple measures (as described in our blog post) and other "gold standards" to gauge the impact of the money spent and effort expended. I would urge us to instead support greater experimentation, smaller projects, more coordination and better communication. We should surely measure indicators of negative effects--blood tests to measure what may be going wrong (or right) with our bodies are very useful for instance.

We also need to consider that every change usually has multiple effects, some positive and others negative. We must constantly look for additional side effects and dynamically adapt whatever we do. Sticking with our obesity example, there is evidence that high fat, low sugar diets, generally known as ketogenic diets, are great for losing weight and preventing diabetes; the improvement can be assessed by measuring one's blood glucose levels. However, recent studies show that this diet might contribute to thyroid problems and if we adhere to one, we must monitor thyroid function and occasionally take breaks from it.

Coming up with hypotheses about causal relationships, testing them and connecting them to larger complex models of how we think the world works is an important step. In addition, asking whether we are asking the right questions and solving the right problems, rather than prematurely focusing on solutions, is key. Jed Emerson, who pioneered early attempts to monetize the economic value of social impact, makes the same point in his recent book The Purpose of Capital.

For the last 1,300 years, the Ise Shrine in Japan has been ritually rebuilt by craftspeople every 20 years. The lumber mostly comes from the shrine's forest managed in 200 year time scales as part of a national afforestation plan dating back centuries. The number of people working at Ise Shrine isn't growing, the shrine isn't trying to expand its business, and its workers are happy and healthy--the shrine is flourishing. Their primary concern is the resilience of the forest, rivers, and natural environment around the shrine. How would we measure their success and what can we learn from their flourishing as we try to manage our society and our planet?

It is heartening to see impact investors developing evidence-based methods to tackle the complex and critical challenges that face us. It's also heartening that capital markets and investors are supportive of investing, and in some cases even accepting reduced returns, in an effort to help tackle our big, complex challenges. We must, however, make changes in the way we fund potential solutions so that it supports a diversity of disciplines and approaches. That, in turn will require new methods of measurement and perhaps we can take advantage of some very old ones, such as the data from Shinto priests who have been measuring ice on a lake for resist oversimplification. If we don't, we risk wasting these funds or, even worse, amplifying existing problems and creating new ones.

Ethan Zuckerman thoughtfully and appropriately points out that one big missing question in my recent Wired piece on measuring philanthropic impact is whether some of this positive societal change should be in the hands of government instead of philanthropists. He correctly points out that since the Reagan/Thatcher era of the 80s, we've started shrinking the role of government and have started to see big philanthropists and the private sector being called on to do what government used to do. In a post from 2013, Ethan wonders why he doesn't have rail solution to his commuting problem from Western Massachusetts. He suggests that without government, things like railway system are difficult to fund - the market isn't the best solution for many social goods.

I think the idea about whether we should be doubling down on philanthropy or fixing government and increasing government resources is a great question and probably the right one. I think the idea of fixing the government and turning the corner on the privatization is a daunting idea, but something we need to discuss.

I decided to write my column this month in Wired about impact investing and the opportunity to bring new perspectives to the space. As I wrote the piece and started to negotiate with my truly great editor at Wired, I got feedback that it was a bit dense, jargony and wonky. My colleague Louis Kang was doing a lot of research for the article, so I decided to move the nitty-gritty details from the Wired piece to this co-authored "explainer" essay. This essay, now a companion piece to the Wired column, is an overview of what impact investing is, describing some different ways that we currently measure impact and some of the concerns we have with these measurement methods. The Wired article discusses my observations of this field and provides some suggestions on how we might better measure impact.

- Joi

Impact Investment Metrics and Their Limitations

By Joi Ito and Louis Kang

As the pile of philanthropic money aimed at solving the world's problems grows, the desire for assessment and rigor has pushed experts to develop metrics to measure impact and success.

But our world's biggest problems -- climate change, poverty, global health, social instability -- don't easily lend themselves to measurement. Climate change, poverty, global health and social instability, for instance, are complex self-adaptive systems that are irreducible to simple metrics and mathematics. In fact, it's simple math and the hyper-efficient optimizations of the financial markets that have caused most of these problems in the first place. Consider for example, capital markets that focus much more on shareholders than other stakeholders, which has caused extraction and exploitation of natural resources; the efficient production of cheap calories that has contributed to obesity; mass consumption that has led to climate change; and Internet and social media platforms that have amplified hate speech and new forms of adversarial attacks. Are modern foundations and financial institutions armed with quants and global development principles, such as the UN's Sustainable Development Goals, enough to tackle such complex challenges? I don't think so.

Philanthropy as a concept has existed for centuries.The U.S. Internal Revenue Service began providing tax benefits for charitable gifts in the early 1900s, and since then, philanthropy has continued to grow and become more sophisticated.

At the MacArthur Foundation, where I serve on the Board of Directors, "impact investing" emerged in the early 1980s as a way to channel capital to communities plagued by underinvestment and spur the growth of revenue-generating nonprofits and social-purpose businesses. Around this time, Nobel Peace Prize winner Muhammad Yunus founded the Grameen Bank on the principle that loans are more effective than charity to disrupt poverty, and it started by offering tiny loans to impoverished entrepreneurs, which we now know as microfinance. Since then, new types of investment capital and assets, as well as financing and organizational structures and impact measurement practices, have emerged to better engage in the active creation of positive impact. Although the purpose and practice of impact investing are continuously revisited and refined, the core idea is to unlock more traditional investment capital to contribute to solving the world's problems. Today, more than 1,340 organizations manage roughly $500 billion in impact investing assets worldwide.

Many companies now proactively claim to be public benefit companies or are undergoing certification by B-Lab to qualify as B-Corps. These include Patagonia and a company that I invested in, Kickstarter. These companies claim to use, and sometimes disclose, auditable measures of their non-financial societal impact. In addition to companies like these, there is a push among more mainstream businesses to go beyond mere measures of financial success and assess their societal or environmental impacts with a "triple bottom line." Although impact investing has largely been seen as a philanthropic activity, which by definition is prone to accepting little or no return on investment, many traditional impact funds and investors now assert that they are designing investment practices to achieve market level returns on investments and meet positive impact targets. According to one Global Impact Investing Network (GIIN) report, 49 such funds have, on average, achieved an 18.9 percent return on equity-based impact investments in emerging markets. Recently, we've seen more established institutional investors, such as Goldman Sachs, KKR and Bain Capital, to name a just a few now active in the impact investing scene.

Texas Pacific Group (TPG) has created an impact investment fund called the Rise Fund with the help of The Bridgespan Group. The Rise Fund has devised a method that attempts to calculate the economic value of impact called the Impact Multiple of Money, or IMM. IMM is one of a growing number of models and protocols, each of which comes with pros and cons, used to assess non-financial impact. The Rise/Bridgespan method generates an economic estimate of the social impact of an investment by first estimating the number of people impacted by it using relevant scientific studies and multiplying that number by the U.S. "value of life" of $5.4 million, as calculated by the U.S. Department of Transportation to quantify "the additional cost that individuals would be willing to bear for improvements in safety (that is, reductions in risks)." This dollar value of the investment's impact is then adjusted by multiplying it by something called the "probability of impact realization," which is an estimated probability of achieving the expected impact calculated based on a review of relevant scientific studies. Using this number, Rise then projects the investment's Net Present Value, or NPV, using an estimated annual discount set by itself. Finally, the NPV is multiplied by the percent of the company's overall equity owned by Rise to figure out how much of the impact Rise is accountable for, which is then divided by the investment amount to determine the IMM (see this HBR case about an alcoholism program that is part of the Rise Fund as an example). For example, if Rise invested $10 million for 50 percent of the equity in a venture, when the NPV is $100 million, Rise determines $50M ($100 million multiplied by 50 percent) is the value of the impact for which it can claim credit. So its IMM would be five times its investment, or $50 million divided by $10 million, the amount it spent to make the investment. In this example, the IMM was five times its investment, exceeding the three times minimum IMM for the Rise Fund.

Robin Hood, which claims to be New York's largest poverty-fighting organization, takes a similar approach to the IMM. It uses a Benefit-Cost Ratio (BCR) to "assign a dollar figure to the amount of philanthropic good that a grant does" and focuses solely on improving the Quality-Adjusted Life Year (QALY). Robin Hood's metrics are demonstrated over 163 different cases, which can be found here. For example, the BCR of Robin Hood's support for a substance abuse treatment program was calculated by first counting the number of individuals who received the treatment as reported by their grantee. Robin Hood staff then estimated three factors: what percent of these individuals received the treatment solely because of their support; how much the QALY was reduced due to substance abuse, and how much the QALY was improved due to intervention. Suppose the treatment program reached 1,000 people, and Robin Hood estimates that it is only accountable for 10 percent of them. Of those 100 people, QALY reduction due to substance abuse is 10 percent and QALY improvement due to intervention is 20 percent. Robin Hood multiplies 100 by 10 percent, then by 20 percent and finally $50,000 (the QALY value as determined by its staff) to argue that the BCR of the program is approximately $100,000.

Not all of the new approaches attempt to measure impact using a dollar value. My college classmate and collaborator on many projects, Pierre Omidyar, has been an influential leader in impact investing through the work of his organization, the Omidyar Network (ON). The ON funds companies and intermediaries that also provide some social benefit. Over the years, the ON has developed and articulated a variety of methodologies to describe how it measures and categorizes opportunities and risks in funding socially beneficial companies. It has also recently experimented with Acumen's "lean data" approach which seeks to allow rapid iteration in social enterprises in the same way start-ups iterate. Acumen has developed software tools to survey beneficiaries of impact investments and calculate an average Net Promoter Score (NPS), which reflects a combination of many factors. NPS is a method originally developed to measure customer satisfaction in marketing. Via Acument's platform, the ON surveyed 36 investees and 11,500+ customers the investees reached across 18 countries, meaning it received an NPS score of 42 (For comparison, Apple's NPS is 72). And with its surveying capability, the ON contends that its investments improved the quality of life of about 74 percent of its customers.

Having reviewed various impact measurement techniques that are practiced today, now ask yourself: IMMs, BCRs, and NPSs -- do these numbers truly reflect what impact means? Understanding impact through measurement has importance, but we must be careful not to oversimplify complex systems into reducible metrics and lose sight of the intricate dynamics of the world. "Of course, many of us who want to see impact investing attain real scale would welcome the simplicity of an 'Impact Earnings Per Share' calculation or other simplified way to compare relative impact of competing investment opportunities -- but being able to advance a simple metric and having a metric framework that actually helps us assess our true impact and value contribution are two different things," Jed Emerson, who invented the framework Social Return on Investment (SROI), recently told me. "As we know from the history of economics and finance, a single metric can't reflect more nuanced aspects of value or impact. Simplified metrics tell us how we are thinking today but not what we truly seek to know."

Impact measurement is still a nascent field. Understanding impact is fragmented, sometimes misguided, and often inadequate. This makes evaluating and generating impact highly inefficient. We need more clarity and transparency, as well as robust scholarship to study and maximize the impact of philanthropy and impact investing. To address this, I've started to discuss and develop new methods to measure impact. My early thoughts and suggestions are introduced in this upcoming Wired article.

Space Exploration and the Age of the Anthropocosmos »

The new generation of space architects can learn from our successes and failures in protecting the internet commons and build a better paradigm for space.

It's OK That Amazon Will (Likely) Get the .amazon Domain »

Based on my time at ICANN, I know how hard it was to make the right decisions in the face of what, to the public, appeared to be obviously wrong.

Optimize Algorithms to Support Kids Online, Not Exploit Them »

Young people benefit from their online interactions. Locking them out of the internet isn’t the answer to commercial bad actors.

My talk at the MIT-Harvard Conference on the Uyghur Human Rights Crisis »

A few weeks ago I was asked to make some remarks at the MIT-Harvard Conference on the Uyghur Human Rights Crisis. When the student organizer of the conference, Zuly Mamat, asked me to speak at the event, I wasn't sure what I would say because I'm definitely not an expert on this topic. But as I dove into researching what is happening to the Uyghur community in China, I realized that it connected to a lot of the themes I have run up against in my own work, particularly the importance of considering the ethical and social implications of technology...

2019 Applied Ethical and Governance Challenges in AI - Notes from Part 3: Intervention »

Applied Ethical and Governance Challenges in Artificial Intelligence (AI) Part 3: Intervention We recently completed the third and final section of our course that I co-taught with Jonathan Zittrain and TA'ed by Samantha Bates, John Bowers and Natalie Saltiel. The plan was to try to bring the discussion of diagnosis and prognosis in for a landing and figure out how to intervene. The first class of this section (the eighth class of the course) looked at the use of algorithms in decision making. One paper that we read was the most recent in a series of papers by Jon Kleinberg,...

I Embraced Screen Time With My Daughter--and I Love It »

Everyone frets about screen time, but what they should really be focusing on is something called connected parenting.

2019 Applied Ethical and Governance Challenges in AI - Notes from Part 2: Prognosis »

This is the second of three parts of the syllabus and summaries prepared by Samantha Bates who TAs the Applied Ethical and Governance Challenges in Artificial Intelligence course which I co-teach with Jonathan Zittrain. John Bowers and Natalie Satiel are also TAs for the course. I posted Part I earlier in the month. My takeaways: In Part I, we defined the space and tried to frame and understand some of the problems. We left with concerns about the reductionist, poorly defined and oversimplified notions of fairness and explainability in much of the literature. We also left feeling quite challenged by...

2019 Applied Ethical and Governance Challenges in AI - Notes from Part 1: Diagnosis »

Jonathan Zittrain and I are co-teaching a class together for the third time. This year, the title of the course is Applied Ethical and Governance Challenges in Artificial Intelligence. It is a seminar, which means that we invite speakers for most of the classes and usually talk about their papers and their work. The speakers and the papers were mostly curated by our amazing teaching assistant team - Samantha Bates, John Bowers and Natalie Satiel. One of the things that Sam does is help prepare for the class by summarizing the paper and the flow of the class and I...

Supposedly 'Fair' Algorithms Can Perpetuate Discrimination »

How the use of AI runs the risk of re-creating the insurance industry's inequities of the previous century.