What is impact evaluation?

The word “evaluation” can be interpreted in many different ways. It means different things to different people and organizations. Engineers, for example, might evaluate or test the quality of a product design, the durability of a material, efficiency of a production process, or the safety of a bridge. Critics evaluate or review the quality of a restaurant, movie or book. A child psychologist may evaluate or assess the decision-making process of toddlers.

At Ethos we evaluate performance of marketing and communication projects. Our researches are designed to improve performance and management. Yes, we use randomized evaluation,  and impact evaluation to achieve a broad understanding of the real EVALUATION of a project.

At a very basic level, randomized evaluation can answer the question: was the project effective? But if thoughtfully designed and implemented, it can also answer the questions, how effective was it? Were there unintended side-effects? Who benefited most? Who was harmed? Why did it work or not work? What lessons can be applied to other contexts, or if the project was scaled up? How cost-effective was it? How does it compare to other programs designed to accomplish similar goals?

In this case, following the framework on comprehensive evaluations we offer:

1. Needs Assessment
2. Process Evaluation
3. Impact Evaluation
4. Cost-Benefit, Cost-Effectiveness, and Cost-Comparison Analysis
5. Goals, Outcomes, and Measurement

Random Acts ::: What happens when you approach global poverty as a science experiment?

This article was published on Slate last month. It discusses the importance of randomized control trials in public policy impact measurement.

By Joshua Keating

If you want to help people start a business, it’s better to give them a loan than a handout. If you want to prevent the spread of HIV and unwanted pregnancies, teach people about safe sex and make condoms widely available. If you want students to learn, put them in smaller classes with new textbooks.

These are all reasonable assumptions based on what we know about economics and human nature. But none stand up—or at least, none are as simple as they seem—when subjected to the kind of randomized controlled trials that have recently been transforming the way we think about economic development and aid to poor countries. Is randomization as revolutionary as its proponents claim? Or is it just another illusory fix to the intractable problem of global poverty?

The idea behind randomized controlled trials should be familiar to any high school science student. In order to test the effect of a variable on a given subject, you need a control group where the variable is not present. In a medical trial, this could mean giving a new drug to one group of patients and a placebo to another. In development, it could mean giving loans and business training to the poorest women in one group of villages in a given area, but not another group, and studying what happens over the next few months. 

Nongovernmental organizations and governments have been slow to adopt the idea of testing programs to help the poor in this way. But proponents of randomization—“randomistas,” as they’re sometimes called—argue that many programs meant to help the poor are being implemented without sufficient evidence that they’re helping, or even not hurting.

Economist Michael Kremer, now the Gates Professor of Developing Societies at Harvard University, carried out one of the earliest randomized development experiments to gain widespread attention, in western Kenya in the early 1990s. Schools in the area had a chronic shortage of textbooks; there was a consensus that more were needed to improve educational outcomes. But when 25 schools out of 100 were randomly chosen to receive new textbooks, they showed little change in average test scores—the only students whose performance did seem to improve were those already at the top of their classes.

The problem likely had to do with language: School is taught in English in Kenya, but for most students it’s a third language behind Swahili and local languages. New textbooks might help those who already speak English well, but for the majority of students, they make little difference without new ways of teaching. Another study by Kremer and Edward Miguel found that providing students in Kenyan schools with deworming medicine not only improved health outcomes but also decreased rates of absenteeism by one quarter. 

The gospel of randomized controlled trials, or RCTs, has spread significantly since Kremer’s early studies. “I’m not suggesting that every organization in every situation should be doing this,” Kremer told Slate. “The vast majority of development projects are not subject to any evaluation of this type, but I’d argue the number should at least be greater than it is now.”

According to Abhijit Banerjee, the Ford Foundation International Professor of Economics at MIT, part of the reason NGOs and governments have been slow to adopt RCTs is practical: They’re usually slow and expensive to carry out, particularly in the midst of scarce resources and clear humanitarian need. The idea of conducting “experiments” on the poor also makes people nervous, for obvious reasons. “When you use the word experiment, people have a vision that these are rats in cages and you’re feeding them some strange chemicals and seeing what happens,” Banerjee told Slate. “That perception has changed.”

“Every development program that’s being done is an experiment,” says Chris Blattman, a political scientist at Columbia University and author of a popular blog on international development. “The funny thing about most aid programs is how nontransparent they are.”

If Kremer was the pioneer of this type of research, Banerjee and his MIT colleague Esther Duflo, co-founders of the Abdul Latif Jameel Poverty Action Lab, have been its most effective evangelists. Their co-authored 2012 book Poor Economics: A Radical Rethinking of the Way to Fight Global Poverty helped introduce the idea of RCTs to the wider public; Duflo has delivered a popular TED talk on the subject and has beenprofiled by The New Yorker.

Some of the examples in Poor Economics seem perfectly tailored for the Malcolm Gladwell–Freakonomics era of counterintuitive pop social science. One RCT carried out in Kenya by Duflo, Kremer, and Stanford University’s Pascaline Dupas found that providing teenage girls with school uniforms reduced rates of teen pregnancy to a greater extent than sex-ed programs. Experts had assumed that a lack of knowledge was the reason girls were having unprotected sex. But often girls with limited opportunities were choosing to get pregnant; giving them uniforms made it easier for them to stay in school instead.

The RCT scalpel has also targeted some of international development’s sacred cows. Take microfinance: the idea of giving small, low-interest loans to the world’s poorest people—particularly women—as a means of alleviating extreme poverty. Microfinance earned Bangladeshi economist Muhammad Yunus and his Grameen Bank the Nobel Peace Price in 2006. But the randomistas have not been impressed; studies in India, Mongolia, Morocco, and the Philippines show that while such loans can help small business start up, it does little to reduce long-term poverty or to empower women. Similarly, a randomized study from Peru on the much-touted “one laptop per child” program, which gave cheaply produced laptops to schoolchildren in poor countries, did little to help students’ academic performance, as the kids were mostly using them for nonschool activities.   

In 2011, Kenyan economist Bernadette Wanjala conducted a covert evaluation of Millennium Villages, the showcase communities provided with health, agriculture, and educational aid as part of an anti-poverty effort spearheaded by Columbia economist Jeffrey Sachs. When Wanjala and her collaborators took surveys of houses that had and had not been given Millennium Village aid packages, they found that it did increase agricultural yields but also “caused less diversification of household economic activity into profitable non-farm employment, tending to decrease household income.” In other words, the people in the village may have been eating better, but weren’t getting richer.

The randomistas insist that they’re simply adding more rigor to the kind of natural experiments that happen in any intervention. But any time well-funded western research institutions start interfering in the lives of the world’s poor, ethical concerns are certain to follow.

Media and policymakers tend to overstate the conclusions of randomized controlled trials.

“Creating a controlled environment to answer these questions via a randomized clinical trial may remove the context from the research, leaving us knowing the true answer without knowing what the right answer is,” physician Paul Farmer wrote in 2012. “Stripping away context, both local and translocal, creates the illusion of equipoise in a world riven by poverty and social disparities.” Angus Deaton, a Princeton University development economist who has emerged as one of the most persistent and outspoken RCT critics, argues that it’s almost impossible to hold conditions entirely constant. What’s more, an intervention that works in say, Kenya, could have a completely different impact in Bangladesh or Haiti.

But the problem of generalizability is hardly confined to RCTs. “Let’s say there was an anti-crime program that worked in Philadelphia, and we were trying to decide if we were going to do it in Boston,” Kremer says. “Of course you have to think about the differences between the two programs, and the two social situations. But you have to think about those things whether it’s a randomized evaluation or a before-and-after study.” While no program evaluation is perfect, RCTs at least do more to eliminate potential confounding variables.

Part of the problem is that media and policymakers tend to overstate the conclusions of researchers conducting RCTs. Banerjee, for instance, feels that the case against microcredit can be overstated. “People often read it as saying that microcredit doesn’t work,” he said. “But the microcredit experiments basically show that if I take someone who has no business experience, [a loan] doesn’t help their income. That doesn’t mean that there aren’t experienced businesspeople who could be helped by a loan.”

Randomization may be good at evaluating small interventions at the level of one village, but if it’s really going to be useful in tackling poverty at a macro level, it needs to tell us something about national-level policies—things like taxation, trade policy, and distribution of natural resource profits.

Researchers are now beginning to apply RCTs to some previously untouched areas of public policy. One recent study, for instance, looked at how monitoring can reduce corruption in Indonesian road building projects. Another looked at how local elections impacted governance in Afghanistan. The success of studies like these may go a long way toward determining whether RCTs turn out to have an impact beyond academia.

Randomization has undoubtedly improved scholarship on development work, imbuing economic studies of interventions in the developing world with more methodological rigor. But whether it can actually contribute meaningfully to reductions in global poverty is another question. As the economist Lant Pritchettrecently wrote, commemorating the 10th anniversary of the Poverty Action Lab, “The argument that RCTs would be more than a tiny component of the overall process of improving development outcomes seems, even now, 10 years in, at best not provable and at worst not very likely.”

That seems a bit pessimistic, but it’s a challenge the randomistas need to answer. For now, the data’s not quite in yet.

Ethos recommends: Mobilities in cities

In April 11th, the fifth annual conference on cities and modern urban realities will bring together scholars from architecture, civil engineering, sociology, and other disciplines, with practitioners whose work addresses the acute issues of urban life. In particular focus will be the subject of mobilities, including visible ones like cars, bikes, walking and crowds, as well as invisible ones like digital and mechanical networks.

Our Project Manager and Clumbia - SIPA student Bruna Santos will be there! Mobilities in Cities is hosted by Saskia Sassen, Co-chair of the Committee on Global Thought. 
This conference is part of a larger project on Urbanizing Technology supported by the Audi Urban Future Initiative.

Program

10:00am: Urbanizing Technology. The Audi Urban Future Initiative

Saskia Sassen, Robert S. Lynd Professor of Sociology and Co-chair, Committee on Global Thought, Columbia University

10:30am-12:00pm Today’s Mobility Spaces and Beyond

Urban Form, Transport Networks and Travel Demand: The Accessibility Nexus
Eric J. Miller, Professor, Department of Civil Engineering and Director, University of Toronto Transportation Research Institute, University of Toronto

Comparative Political Economy of Sustainable Transport
Walter Hook, Chief Executive Officer, Institute for Transportation & Development Policy, New York

Moving beyond the ‘Car’
John Urry, Author of Societies beyond Oil: Oil Dregs and Social Futures (Zed 2013) and Offshoring (Polity 2014)

12:15pm-1:15pm Transmobilities

Toward Transmobility
Adam Greenfield, Founder and managing director, Urbanscale and Senior Urban Fellow, LSE Cities

The Future of Urban Mobility in China
Xuefei Ren, Associate Professor of Sociology and Global Urban Studies, Michigan State University

1:15pm-2:15pm Lunch Break

2:15pm-4:00pm Measuring, Making, Subverting Urban Futures

What does the Global Power City Index 2013 tell us about Mobility in New York, Tokyo and two other cities?
Hiroo Ichikawa, Dean, Graduate School of Governance Studies, Meiji University and Executive Director, Mori Memorial Foundation, Japan

Moving Bits
Assaf Biderman, Associate Director, SENSEable City Lab, Massachusetts Institute of Technology

Bogotá: Urban Space for a New Mobility
Diana Barco, Co-curator of the Audi Urban Future Initiative Bogotá project on mobility and urban fabric
Doris Tarchópulos, Co-curator and Co-director of the research, “Bogotá Urban Interactions and Future Mobility,” Audi Urban Future Initiative

4:15pm-5:45pm Programming Mobility

Reprogramming Mobility
Greg Lindsay, Senior Fellow, World Policy Institute

Modes of Mobility: Restricted, Imposed, Ignored, Opened or Hacked
Claire Roberge, Researcher, Urbanizing Technology: The Mobility Complex

How Motion Came to Dominate Urban Space
Richard Sennett, University Professor of Humanities, New York University and Professor of Sociology, London School of Economics and Political Science

5:45pm Closing Remarks: What is next in this research project?

Saskia Sassen

Sponsored by the Committee on Global Thought and the Graduate School of Architecture, Planning and Preservation.

The event is free and open to the public. No registration necessary. First come first seated.

For more information on this and other events, please visit cgt.columbia.edu