[go: up one dir, main page]
IDEAS home Printed from https://ideas.repec.org/p/cpr/ceprdp/6059.html
   My bibliography  Save this paper

Using Randomization in Development Economics Research: A Toolkit

Author

Listed:
  • Kremer, Michael
  • Glennerster, Rachel
  • Duflo, Esther
Abstract
This paper is a practical guide (a toolkit) for researchers, students and practitioners wishing to introduce randomization as part of a research design in the field. It first covers the rationale for the use of randomization, as a solution to selection bias and a partial solution to publication biases. Second, it discusses various ways in which randomization can be practically introduced in a field settings. Third, it discusses designs issues such as sample size requirements, stratification, level of randomization and data collection methods. Fourth, it discusses how to analyze data from randomized evaluations when there are departures from the basic framework. It reviews in particular how to handle imperfect compliance and externalities. Finally, it discusses some of the issues involved in drawing general conclusions from randomized evaluations, including the necessary use of theory as a guide when designing evaluations and interpreting results.

Suggested Citation

  • Kremer, Michael & Glennerster, Rachel & Duflo, Esther, 2007. "Using Randomization in Development Economics Research: A Toolkit," CEPR Discussion Papers 6059, C.E.P.R. Discussion Papers.
    • Handle: RePEc:cpr:ceprdp:6059
    as

    Download full text from publisher

    File URL: https://cepr.org/publications/DP6059
    Download Restriction: CEPR Discussion Papers are free to download for our researchers, subscribers and members. If you fall into one of these categories but have trouble downloading our papers, please contact us at subscribers@cepr.org
    ---><---

    As the access to this document is restricted, you may want to look for a different version below or search for a different version of it.

    Other versions of this item:

    References listed on IDEAS

    as
    1. Abhijit V. Banerjee & Shawn Cole & Esther Duflo & Leigh Linden, 2007. "Remedying Education: Evidence from Two Randomized Experiments in India," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 122(3), pages 1235-1264.
    2. James J. Heckman & Edward Vytlacil, 2005. "Structural Equations, Treatment Effects, and Econometric Policy Evaluation," Econometrica, Econometric Society, vol. 73(3), pages 669-738, May.
    3. Astrid Grasdal, 2001. "The performance of sample selection estimators to control for attrition bias," Health Economics, John Wiley & Sons, Ltd., vol. 10(5), pages 385-398, July.
    4. Bertrand, Marianne & Karlan, Dean S. & Mullainathan, Sendhil & Shafir, Eldar & Zinman, Jonathan, 2005. "What's Psychology Worth? A Field Experiment in the Consumer Credit Market," Center Discussion Papers 28441, Yale University, Economic Growth Center.
    5. Dean Karlan & Jonathan Zinman, 2010. "Expanding Credit Access: Using Randomized Supply Decisions to Estimate the Impacts," The Review of Financial Studies, Society for Financial Studies, vol. 23(1), pages 433-464, January.
    6. Caroline M. Hoxby, 2003. "The Economics of School Choice," NBER Books, National Bureau of Economic Research, Inc, number hox03-1.
    7. Marianne Bertrand & Simeon Djankov & Rema Hanna & Sendhil Mullainathan, 2006. "Does corruption produce unsafe drivers?," Natural Field Experiments 00218, The Field Experiments Website.
    8. Duflo, Esther & Hanna, Rema, 2005. "Monitoring Works: Getting Teachers to Come to School," CEPR Discussion Papers 5426, C.E.P.R. Discussion Papers.
    9. Duflo, Esther & Dupas, Pascaline & Kremer, Michael & Sinei, Samuel, 2006. "Education and HIV/AIDS prevention : evidence from a randomized evaluation in Western Kenya," Policy Research Working Paper Series 4024, The World Bank.
    10. Angrist, Joshua D, 1990. "Lifetime Earnings and the Vietnam Era Draft Lottery: Evidence from Social Security Administrative Records," American Economic Review, American Economic Association, vol. 80(3), pages 313-336, June.
    11. Maluccio, John A. & Flores, Rafael, 2004. "Impact evaluation of a conditional cash transfer program," FCND briefs 184, International Food Policy Research Institute (IFPRI).
    12. Glenn W. Harrison & John A. List, 2004. "Field Experiments," Journal of Economic Literature, American Economic Association, vol. 42(4), pages 1009-1055, December.
    13. Esther Duflo & Emmanuel Saez, 2003. "The Role of Information and Social Interactions in Retirement Plan Decisions: Evidence from a Randomized Experiment," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 118(3), pages 815-842.
    14. Imbens, Guido W & Angrist, Joshua D, 1994. "Identification and Estimation of Local Average Treatment Effects," Econometrica, Econometric Society, vol. 62(2), pages 467-475, March.
    15. Michael Kremer & Edward Miguel, 2003. "Networks, social learning, and technology adoption: The case of deworming drugs in kenya," Natural Field Experiments 00312, The Field Experiments Website.
    16. Joshua Angrist & Eric Bettinger & Erik Bloom & Elizabeth King & Michael Kremer, 2002. "Vouchers for Private Schooling in Colombia: Evidence from a Randomized Natural Experiment," American Economic Review, American Economic Association, vol. 92(5), pages 1535-1558, December.
    17. Caroline Minter Hoxby, 2003. "School Choice and School Productivity. Could School Choice Be a Tide that Lifts All Boats?," NBER Chapters, in: The Economics of School Choice, pages 287-342, National Bureau of Economic Research, Inc.
    18. Nava Ashraf & Dean Karlan & Wesley Yin, 2006. "Tying Odysseus to the Mast: Evidence From a Commitment Savings Product in the Philippines," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 121(2), pages 635-672.
    19. Banerjee, Abhijit & Bardhan, Pranab & Basu, Kaushik, 2005. "New Directions in Development Economics: Theory or Empirics? A Symposium in Economic and Political Weekly," Working Papers 127128, Cornell University, Department of Applied Economics and Management.
    20. Glewwe, Paul & Kremer, Michael & Moulin, Sylvie & Zitzewitz, Eric, 2004. "Retrospective vs. prospective analyses of school inputs: the case of flip charts in Kenya," Journal of Development Economics, Elsevier, vol. 74(1), pages 251-268, June.
    21. Parker, Susan W. & Rubalcava, Luis & Teruel, Graciela, 2008. "Evaluating Conditional Schooling and Health Programs," Handbook of Development Economics, in: T. Paul Schultz & John A. Strauss (ed.), Handbook of Development Economics, edition 1, volume 4, chapter 62, pages 3963-4035, Elsevier.
    22. Mark M. Pitt & Shahidur R. Khandker, 1998. "The Impact of Group-Based Credit Programs on Poor Households in Bangladesh: Does the Gender of Participants Matter?," Journal of Political Economy, University of Chicago Press, vol. 106(5), pages 958-996, October.
    23. Petra E. Todd & Kenneth I. Wolpin, 2008. "Ex Ante Evaluation of Social Programs," Annals of Economics and Statistics, GENES, issue 91-92, pages 263-291.
    24. Das, J. & Dercon, S. & Habyarimana, J. & Krishnan, P., 2004. "‘When Can School Inputs Improve Test Scores?’," Cambridge Working Papers in Economics 0437, Faculty of Economics, University of Cambridge.
    25. Dean Karlan & Jonathan Zinman, 2009. "Observing Unobservables: Identifying Information Asymmetries With a Consumer Credit Field Experiment," Econometrica, Econometric Society, vol. 77(6), pages 1993-2008, November.
    26. repec:feb:natura:0021 is not listed on IDEAS
    27. A. Smith, Jeffrey & E. Todd, Petra, 2005. "Does matching overcome LaLonde's critique of nonexperimental estimators?," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 305-353.
    28. Marianne Bertrand & Dean Karlan & Sendhil Mullainathan & Eldar Shafir & Jonathan Zinman, 2010. "What's Advertising Content Worth? Evidence from a Consumer Credit Marketing Field Experiment," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 125(1), pages 263-306.
    29. Ravallion, Martin, 2008. "Evaluating Anti-Poverty Programs," Handbook of Development Economics, in: T. Paul Schultz & John A. Strauss (ed.), Handbook of Development Economics, edition 1, volume 4, chapter 59, pages 3787-3846, Elsevier.
    30. Paul Glewwe & Nauman Ilias & Michael Kremer, 2010. "Teacher Incentives," American Economic Journal: Applied Economics, American Economic Association, vol. 2(3), pages 205-227, July.
    31. Glewwe, Paul & Kremer, Michael, 2006. "Schools, Teachers, and Education Outcomes in Developing Countries," Handbook of the Economics of Education, in: Erik Hanushek & F. Welch (ed.), Handbook of the Economics of Education, edition 1, volume 2, chapter 16, pages 945-1017, Elsevier.
    32. Steven Glazerman & Dan M. Levy & David Myers, "undated". "Nonexperimental Replications of Social Experiments: A Systematic Review," Mathematica Policy Research Reports 9e8a2edf60ef4fa89ad9a5acf, Mathematica Policy Research.
    33. De Long, J Bradford & Lang, Kevin, 1992. "Are All Economic Hypotheses False?," Journal of Political Economy, University of Chicago Press, vol. 100(6), pages 1257-1272, December.
    34. Edward Miguel & Michael Kremer, 2004. "Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities," Econometrica, Econometric Society, vol. 72(1), pages 159-217, January.
    35. James Heckman & Hidehiko Ichimura & Jeffrey Smith & Petra Todd, 1998. "Characterizing Selection Bias Using Experimental Data," Econometrica, Econometric Society, vol. 66(5), pages 1017-1098, September.
    36. Joshua D. Angrist, 1998. "Estimating the Labor Market Impact of Voluntary Military Service Using Social Security Data on Military Applicants," Econometrica, Econometric Society, vol. 66(2), pages 249-288, March.
    37. Michael Kremer & Edward Miguel & Rebecca Thornton, 2009. "Incentives to Learn," The Review of Economics and Statistics, MIT Press, vol. 91(3), pages 437-456, August.
    38. Ashenfelter, Orley & Harmon, Colm & Oosterbeek, Hessel, 1999. "A review of estimates of the schooling/earnings relationship, with tests for publication bias," Labour Economics, Elsevier, vol. 6(4), pages 453-470, November.
    39. Marianne Bertrand & Esther Duflo & Sendhil Mullainathan, 2004. "How Much Should We Trust Differences-In-Differences Estimates?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 119(1), pages 249-275.
    40. Raghabendra Chattopadhyay & Esther Duflo, 2004. "Women as Policy Makers: Evidence from a Randomized Policy Experiment in India," Econometrica, Econometric Society, vol. 72(5), pages 1409-1443, September.
    41. Karlan, Dean S. & Zinman, Jonathan, 2005. "Elasticities of Demand for Consumer Credit," Center Discussion Papers 28485, Yale University, Economic Growth Center.
    42. Gustavo J. Bobonis & Edward Miguel & Charu Puri-Sharma, 2006. "Anemia and School Participation," Journal of Human Resources, University of Wisconsin Press, vol. 41(4).
    43. Joshua D. Angrist & Guido W. Imbens & D.B. Rubin, 1993. "Identification of Causal Effects Using Instrumental Variables," NBER Technical Working Papers 0136, National Bureau of Economic Research, Inc.
    44. Juan Jose Diaz & Sudhanshu Handa, 2006. "An Assessment of Propensity Score Matching as a Nonexperimental Impact Estimator: Evidence from Mexico’s PROGRESA Program," Journal of Human Resources, University of Wisconsin Press, vol. 41(2).
    45. Paul Schultz, T., 2004. "School subsidies for the poor: evaluating the Mexican Progresa poverty program," Journal of Development Economics, Elsevier, vol. 74(1), pages 199-250, June.
    46. LaLonde, Robert J, 1986. "Evaluating the Econometric Evaluations of Training Programs with Experimental Data," American Economic Review, American Economic Association, vol. 76(4), pages 604-620, September.
    47. James J. Heckman & Hidehiko Ichimura & Petra E. Todd, 1997. "Matching As An Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 64(4), pages 605-654.
    48. David S. Lee, 2002. "Trimming for Bounds on Treatment Effects with Missing Outcomes," NBER Technical Working Papers 0277, National Bureau of Economic Research, Inc.
    49. Dupas, Pascaline, 2005. "Relative Risks and the Market for Sex: Teenagers, Sugar Daddies and HIV in Kenya," MPRA Paper 248, University Library of Munich, Germany, revised Aug 2006.
    50. O. Ashenfelter & D. Card (ed.), 1999. "Handbook of Labor Economics," Handbook of Labor Economics, Elsevier, edition 1, volume 3, number 3.
    51. Joshua Angrist & Eric Bettinger & Michael Kremer, 2006. "Long-Term Educational Consequences of Secondary School Vouchers: Evidence from Administrative Records in Colombia," American Economic Review, American Economic Association, vol. 96(3), pages 847-862, June.
    52. Hausman, Jerry A & Wise, David A, 1979. "Attrition Bias in Experimental and Panel Data: The Gary Income Maintenance Experiment," Econometrica, Econometric Society, vol. 47(2), pages 455-473, March.
    53. Angrist, Joshua D, 1990. "Lifetime Earnings and the Vietnam Era Draft Lottery: Evidence from Social Security Administrative Records: Errata," American Economic Review, American Economic Association, vol. 80(5), pages 1284-1286, December.
    54. Michael Kremer, 2003. "Randomized Evaluations of Educational Programs in Developing Countries: Some Lessons," American Economic Review, American Economic Association, vol. 93(2), pages 102-106, May.
    55. Benjamin A. Olken, 2007. "Monitoring Corruption: Evidence from a Field Experiment in Indonesia," Journal of Political Economy, University of Chicago Press, vol. 115(2), pages 200-249.
    56. Buddelmeyer, Hielke & Skoufias, Emmanuel, 2003. "An Evaluation of the Performance of Regression Discontinuity Design on PROGRESA," IZA Discussion Papers 827, Institute of Labor Economics (IZA).
    57. Pascaline Dupas, 2006. "Relative risks and the market for sex: Teenagers, sugar daddies, and hiv in kenya," Natural Field Experiments 00235, The Field Experiments Website.
    58. Chang-Tai Hsieh & Miguel Urquiola, 2002. "When Schools Compete, How Do They Compete? An Assessment of Chile's Nationwide School Voucher Program," Working Papers 123, Princeton University, Department of Economics, Center for Economic Policy Studies..
    59. Schady, Norbert & Araujo, Maria Caridad, 2006. "Cash transfers, conditions, school enrollment, and child work : evidence from a randomized experiment in Ecuador," Policy Research Working Paper Series 3930, The World Bank.
    60. James J. Heckman & Hidehiko Ichimura & Petra Todd, 1998. "Matching As An Econometric Evaluation Estimator," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 65(2), pages 261-294.
    61. Manski, Charles F., 1989. "Schooling as experimentation: a reappraisal of the postsecondary dropout phenomenon," Economics of Education Review, Elsevier, vol. 8(4), pages 305-312, August.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    2. Abhijit V. Banerjee & Esther Duflo, 2009. "The Experimental Approach to Development Economics," Annual Review of Economics, Annual Reviews, vol. 1(1), pages 151-178, May.
    3. Glewwe, Paul & Kremer, Michael, 2006. "Schools, Teachers, and Education Outcomes in Developing Countries," Handbook of the Economics of Education, in: Erik Hanushek & F. Welch (ed.), Handbook of the Economics of Education, edition 1, volume 2, chapter 16, pages 945-1017, Elsevier.
    4. Justine Burns & Malcolm Kewsell & Rebecca Thornton, 2009. "Evaluating the Impact of Health Programmes," SALDRU Working Papers 40, Southern Africa Labour and Development Research Unit, University of Cape Town.
    5. Guido W. Imbens, 2010. "Better LATE Than Nothing: Some Comments on Deaton (2009) and Heckman and Urzua (2009)," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 399-423, June.
    6. Ravallion, Martin, 2008. "Evaluating Anti-Poverty Programs," Handbook of Development Economics, in: T. Paul Schultz & John A. Strauss (ed.), Handbook of Development Economics, edition 1, volume 4, chapter 59, pages 3787-3846, Elsevier.
    7. David K. Evans & Arkadipta Ghosh, 2008. "Prioritizing Educational Investments in Children in the Developing World," Working Papers 587, RAND Corporation.
    8. Levitt, Steven D. & List, John A., 2009. "Field experiments in economics: The past, the present, and the future," European Economic Review, Elsevier, vol. 53(1), pages 1-18, January.
    9. Esther Duflo, 2005. "Évaluer l'impact des programmes d'aide au développement : le rôle des évaluations par assignation aléatoire," Revue d’économie du développement, De Boeck Université, vol. 13(2), pages 185-226.
    10. David K. Evans & Arkadipta Ghosh, 2008. "Prioritizing Educational Investments in Children in the Developing World," Working Papers WR-587, RAND Corporation.
    11. Felipe Barrera-Osorio & Marianne Bertrand & Leigh L. Linden & Francisco Perez-Calle, 2008. "Conditional Cash Transfers in Education Design Features, Peer and Sibling Effects Evidence from a Randomized Experiment in Colombia," NBER Working Papers 13890, National Bureau of Economic Research, Inc.
    12. Karthik Muralidharan & Paul Niehaus, 2017. "Experimentation at Scale," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 103-124, Fall.
    13. Masino, Serena & Niño-Zarazúa, Miguel, 2016. "What works to improve the quality of student learning in developing countries?," International Journal of Educational Development, Elsevier, vol. 48(C), pages 53-65.
    14. Benjamin A. Olken, 2020. "Banerjee, Duflo, Kremer, and the Rise of Modern Development Economics," Scandinavian Journal of Economics, Wiley Blackwell, vol. 122(3), pages 853-878, July.
    15. Malcolm Keswell & Justine Burns & Rebecca Thornton, 2012. "Evaluating the Impact of Health Programmes on Productivity," African Development Review, African Development Bank, vol. 24(4), pages 302-315.
    16. Angus Deaton, 2009. "Instruments of development: Randomization in the tropics, and the search for the elusive keys to economic development," Working Papers 1128, Princeton University, Woodrow Wilson School of Public and International Affairs, Center for Health and Wellbeing..
    17. Michael Kremer & Edward Miguel & Rebecca Thornton, 2009. "Incentives to Learn," The Review of Economics and Statistics, MIT Press, vol. 91(3), pages 437-456, August.
    18. Todd, Petra E., 2012. "Effectiveness of interventions aimed at improving women's employability and quality of work : a critical review," Policy Research Working Paper Series 6189, The World Bank.
    19. Peter Hull & Michal Kolesár & Christopher Walters, 2022. "Labor by design: contributions of David Card, Joshua Angrist, and Guido Imbens," Scandinavian Journal of Economics, Wiley Blackwell, vol. 124(3), pages 603-645, July.
    20. Eric A. Hanushek, 2008. "Incentives for Efficiency and Equity in the School System," Perspektiven der Wirtschaftspolitik, Verein für Socialpolitik, vol. 9(s1), pages 5-27, May.

    More about this item

    Keywords

    Experiments; Development; Program evaluation;
    All these keywords.

    JEL classification:

    • C93 - Mathematical and Quantitative Methods - - Design of Experiments - - - Field Experiments

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cpr:ceprdp:6059. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://www.cepr.org .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.