Evaluating Microfinance: Randomize!
The UN has declared 2005 as the Year of micro-credit, on the ground that "micro-credit has been changing people's lives and revitalizing communities". However, there have been very few good assessments of the extent to which microfinance has improved the lives of the poor, nor of the best and most beneficial ways to provide financial services to the under-banked.

What is the best way to provide financial services to the poor? What is the impact of microfinance on the socio-economic conditions of these households? Despite plethora of anecdotal evidence, we know very little about these questions. Substantial funds are spent on evaluating microfinance programs. However these evaluations often just evaluate organizational, financial and governance variables within these institutions. The literature has tended to focus on the financial sustainability of microfinance rather than on its developmental impact. While tracking performance at this level is important, we should also be evaluating programs at a more fundamental level to find out whether, for example, providing financial services to the poor increases household's investment capacity and consumption, whether it reduces its vulnerability and whether it increases women's bargaining power within the households. The question of the impact of microfinance, and what are the channels of this impact has remained largely unanswered. While some celebrate micro-credit as the unique solution to poverty alleviation, some argue that it does not have any ground breaking impact on its own.

As a matter of fact, it is very difficult to answer these questions. Imagine, for example, that a person received a micro-loan. Before the program, this person's income was 500 Rs per month, and after the program, it is 1000 Rs. At a first glance, it is tempting to attribute this to the micro-credit program. However, imagine further that in the same time, an income generating program was introduced in the village. The effect we see may be due to this rather than to the microfinance program. To solve this problem, some propose to compare borrowers and non borrowers within the same village. Imagine that a microfinance program is introduced in a village. It is likely that the most entrepreneurial people will be the ones taking up credit. One year after the program, imagine you observe that the person who took up credit is better off than the person who did not. Should one conclude that this difference between both persons is due to the credit program? Well, it is likely that the entrepreneurial person would have been better off in any case. It is therefore usually not appropriate to compare beneficiaries to non beneficiaries to infer the impact of the program. Now imagine that to avoid this caveat we decide to compare those villages that have a microfinance program to villages that do not. However, it is likely that the NGO would have targeted the poorest villages. Therefore, one might observe that the village that did not get micro credit is better off. This would not mean that micro-credit has a negative impact! Inversely, the NGO may have chosen to go to better off villages first, to avoid taking too much risk from the beginning, so that comparing those villages to others would result in an over-estimation of the program effect! The ideal comparison would be to compare Person A in one state of the world (with microfinance) to Person A in another state (without microfinance), at the same time. Of course, this is not possible, but researchers need to create equivalent situations. Dean Karlan has called such a valid control group the "holy grail of any microfinance impact assessment".

Some studies have tried to go around the difficulties we just described, and create a control group, by comparing new entrants to old entrants. Studies using this methodology have become recently quite popular, as they are cheap and easy to implement. For example USAID, through its AIM project, encourages this methodology (Dean Karlan 2001). However, it is important to warn the donors and development community with several caveats. This method presents some advantages compared to the methodology mentioned above, as it considers only people who decided to take up credit, who should therefore be equally entrepreneurial. However, is the set of new clients the same as the set of old clients? No, because some have dropped out in the meantime, either because they were not successful or because they were very successful. Unless one compares all old clients (including the ones who dropped out) to the new clients, this methodology has therefore serious caveats. And following up with the old clients who dropped out is likely to be difficult, making this methodology less easy than it sounds at first. Even if one manages to do that, old clients joined earlier than new clients probably for a reason, and are therefore probably not the same as the new clients. Maybe the "best" clients joined first and the "bad" clients last, convinced by the repeated coming of the loan officer or by mere imitation. In addition, the MFI may have witnessed institutional changes as a result of internal learning so that old and new clients may not have been subject to the same policies. The cleanest and clearest way to establish the impact of microfinance is to run a randomized trial. Choose 100 villages, start a micro-credit program in half of them (randomly chosen), phase in the program later in the other half, and in the meantime compare the outcomes in the two groups of villages. This approach requires that evaluation be built into the design of the original program and that data be collected on all 100 villages, which can be expensive. However, this is what we do if we want to know whether a drug or vaccine is effective, and new research is showing this technique can teach us a lot about development.

Some would argue that such a randomization is unfair or hard to implement in practice. However, there is no organization that starts a program in the same time in all targeted areas: usually, it is possible to introduce randomization by randomly phasing in the order of villages to be brought into the program. Having strong evidence about what works is important for many reasons. Non-governmental organizations (NGOs), MFIs and governments can use this evidence to focus their limited budgets on those programs that are most effective. Following such an evaluation, an MFI may be convinced to expand the same program since it proved to be very effective. The MFI may decide to add some changes into the design as it did not seem to achieve the desired effects. In addition, providing evidence to regulators that microfinance has a strong impact may convince them to mainstream in the regulated system; so far, it has not in India. The evaluation of the micro-credit program of Spandana led by MIT's Abdul Latif Jameel Poverty Action Lab (J-PAL) in collaboration with CMF uses such a randomized trial. The first randomized evaluation of microfinance, it takes advantage of the fact that Spandana was planning to expand into Hyderabad slums. As mentioned above, the evaluation design was done before the programme started. A baseline survey was conducted in 100 slums, among which 50, randomly selected, have started receiving credit. As the 50 slums were randomly selected, they are on average comparable to the 50 remaining slums and the only difference between them should be the micro-credit programme.

A follow-up survey will be conducted after a loan cycle has been completed (i.e. after about a year). After the study is completed, the 50 remaining slums will receive credit as well. Note that the impact of credit may be different after one year and after 2 years. Ideally, the randomized evaluation would last as long as possible. However, there are practical difficulties in doing this, as the comparison slums will be eager to be brought into the program. The study is therefore scheduled to last about 18 months. It is important to highlight that by comparing slums with and without microfinance we will be evaluating the impact of having access to credit, not the impact of taking credit (which for the reasons highlighted above, is not possible). Of course, the results of the Hyderabad-PAL study may not be true for all India, as context may matter. It is therefore essential to carry out other evaluations of micro-credit programs, using the same methods, in other parts of India. MIT's J-PAL, in collaboration with CMF, hopes to conduct similar evaluations in other states and in rural areas.