[go: up one dir, main page]
IDEAS home Printed from https://ideas.repec.org/p/ecl/stabus/3837.html
   My bibliography  Save this paper

Optimal Experimental Design for Staggered Rollouts

Author

Listed:
  • Athey, Susan

    (Stanford Institute for Economic Policy Research)

  • Imbens, Guido W.

    (Stanford Institute for Economic Policy Research)

  • Bayati, Mohsen

    (Stanford University Graduate School of Business)

Abstract
Experimentation has become an increasingly prevalent tool for guiding policy choices, firm decisions, and product innovation. A common hurdle in designing experiments is the lack of statistical power. In this paper, we study optimal multi-period experimental design under the constraint that the treatment cannot be easily removed once implemented; for example, a government or firm might implement treatment in different geographies at different times, where the treatment cannot be easily removed due to practical constraints. The design problem is to select which units to treat at which time, intending to test hypotheses about the effect of the treatment. When the potential outcome is a linear function of a unit effect, a time effect, and observed discrete covariates, we provide an analytically feasible solution to the design problem where the variance of the estimator for the treatment effect is at most 1+O(1/N^2) times the variance of the optimal design, where N is the number of units. This solution assigns units in a staggered treatment adoption pattern, where the proportion treated is a linear function of time. In the general setting where outcomes depend on latent covariates, we show that historical data can be utilized in the optimal design. We propose a data-driven local search algorithm with the minimax decision criterion to assign units to treatment times. We demonstrate that our approach improves upon benchmark experimental designs through synthetic experiments on real-world data sets from several domains, including healthcare, finance, and retail. Finally, we consider the case where the treatment effect changes with the time of treatment, showing that the optimal design treats a smaller fraction of units at the beginning and a greater share at the end.

Suggested Citation

  • Athey, Susan & Imbens, Guido W. & Bayati, Mohsen, 2019. "Optimal Experimental Design for Staggered Rollouts," Research Papers 3837, Stanford University, Graduate School of Business.
    • Handle: RePEc:ecl:stabus:3837
    as

    Download full text from publisher

    To our knowledge, this item is not available for download. To find whether it is available, there are three options:
    1. Check below whether another version of this item is available online.
    2. Check on the provider's web page whether it is in fact available.
    3. Perform a search for a similarly titled item that would be available.

    Other versions of this item:

    References listed on IDEAS

    as
    1. Federico A. Bugni & Ivan A. Canay & Azeem M. Shaikh, 2018. "Inference Under Covariate-Adaptive Randomization," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(524), pages 1784-1796, October.
    2. Susan Athey & Scott Stern, 2002. "The Impact of Information Technology on Emergency Health Care Outcomes," RAND Journal of Economics, The RAND Corporation, vol. 33(3), pages 399-432, Autumn.
    3. Jushan Bai & Serena Ng, 2002. "Determining the Number of Factors in Approximate Factor Models," Econometrica, Econometric Society, vol. 70(1), pages 191-221, January.
    4. DiMasi, Joseph A. & Grabowski, Henry G. & Hansen, Ronald W., 2016. "Innovation in the pharmaceutical industry: New estimates of R&D costs," Journal of Health Economics, Elsevier, vol. 47(C), pages 20-33.
    5. Dmitry Arkhangelsky & Susan Athey & David A. Hirshberg & Guido W. Imbens & Stefan Wager, 2021. "Synthetic Difference-in-Differences," American Economic Review, American Economic Association, vol. 111(12), pages 4088-4118, December.
    6. Varnell, S.P. & Murray, D.M. & Janega, J.B. & Blitstein, J.L., 2004. "Design and Analysis of Group-Randomized Trials: A Review of Recent Practices," American Journal of Public Health, American Public Health Association, vol. 94(3), pages 393-399.
    7. Federico A. Bugni & Ivan A. Canay & Azeem M. Shaikh, 2019. "Inference under covariate‐adaptive randomization with multiple treatments," Quantitative Economics, Econometric Society, vol. 10(4), pages 1747-1785, November.
    8. Susan Athey & Mohsen Bayati & Nikolay Doudchenko & Guido Imbens & Khashayar Khosravi, 2021. "Matrix Completion Methods for Causal Panel Data Models," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 116(536), pages 1716-1730, October.
    9. Lawrie, Jock & Carlin, John B. & Forbes, Andrew B., 2015. "Optimal stepped wedge designs," Statistics & Probability Letters, Elsevier, vol. 99(C), pages 210-214.
    10. Athey, Susan & Imbens, Guido W., 2022. "Design-based analysis in Difference-In-Differences settings with staggered adoption," Journal of Econometrics, Elsevier, vol. 226(1), pages 62-79.
    11. Keisuke Hirano & Guido W. Imbens & Geert Ridder, 2003. "Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score," Econometrica, Econometric Society, vol. 71(4), pages 1161-1189, July.
    12. Li, Fan & Turner, Elizabeth L. & Preisser, John S., 2018. "Optimal allocation of clusters in cohort stepped wedge designs," Statistics & Probability Letters, Elsevier, vol. 137(C), pages 257-263.
    13. Erik Brynjolfsson & Avinash Collis & Felix Eggers, 2019. "Using massive online choice experiments to measure changes in well-being," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 116(15), pages 7250-7255, April.
    14. Leahey, T.M. & Thomas, G. & Fava, J.L. & Subak, L.L. & Schembri, M. & Krupel, K. & Kumar, R. & Weinberg, B. & Wing, R.R., 2014. "Adding evidence-based behavioral weight loss strategies to a statewide wellness campaign: A randomized clinical trial," American Journal of Public Health, American Public Health Association, vol. 104(7), pages 1300-1306.
    15. Murray, D.M. & Varnell, S.P. & Blitstein, J.L., 2004. "Design and Analysis of Group-Randomized Trials: A Review of Recent Methodological Developments," American Journal of Public Health, American Public Health Association, vol. 94(3), pages 423-432.
    16. Li, Kathleen T. & Bell, David R., 2017. "Estimation of average treatment effects with panel data: Asymptotic theory and implementation," Journal of Econometrics, Elsevier, vol. 197(1), pages 65-75.
    17. Alberto Abadie & Alexis Diamond & Jens Hainmueller, 2015. "Comparative Politics and the Synthetic Control Method," American Journal of Political Science, John Wiley & Sons, vol. 59(2), pages 495-510, February.
    18. Nikolay Doudchenko & Guido W. Imbens, 2016. "Balancing, Regression, Difference-In-Differences and Synthetic Control Methods: A Synthesis," NBER Working Papers 22791, National Bureau of Economic Research, Inc.
    19. Dimitris Bertsimas & Nikita Korolko & Alexander M. Weinstein, 2019. "Covariate-Adaptive Optimization in Online Clinical Trials," Operations Research, INFORMS, vol. 67(4), pages 1150-1161, July.
    20. Imbens, Guido W & Angrist, Joshua D, 1994. "Identification and Estimation of Local Average Treatment Effects," Econometrica, Econometric Society, vol. 62(2), pages 467-475, March.
    21. Abadie, Alberto & Diamond, Alexis & Hainmueller, Jens, 2010. "Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program," Journal of the American Statistical Association, American Statistical Association, vol. 105(490), pages 493-505.
    22. Susan Athey & Scott Stern, 1998. "An Empirical Framework for Testing Theories About Complimentarity in Organizational Design," NBER Working Papers 6600, National Bureau of Economic Research, Inc.
    23. Robert P. Leone, 1995. "Generalizing What Is Known About Temporal Aggregation and Advertising Carryover," Marketing Science, INFORMS, vol. 14(3_supplem), pages 141-150.
    24. Sinan Aral & Dylan Walker, 2014. "Tie Strength, Embeddedness, and Social Influence: A Large-Scale Networked Experiment," Management Science, INFORMS, vol. 60(6), pages 1352-1370, June.
    25. Alberto Abadie & Guido W. Imbens, 2006. "Large Sample Properties of Matching Estimators for Average Treatment Effects," Econometrica, Econometric Society, vol. 74(1), pages 235-267, January.
    26. Dimitris Bertsimas & Mac Johnson & Nathan Kallus, 2015. "The Power of Optimization Over Randomization in Designing Experiments Involving Small Samples," Operations Research, INFORMS, vol. 63(4), pages 868-876, August.
    27. Jushan Bai, 2003. "Inferential Theory for Factor Models of Large Dimensions," Econometrica, Econometric Society, vol. 71(1), pages 135-171, January.
    28. Erik Brynjolfsson & Felix Eggers & Avinash Gannamaneni, 2018. "Measuring Welfare with Massive Online Choice Experiments: A Brief Introduction," AEA Papers and Proceedings, American Economic Association, vol. 108, pages 473-476, May.
    29. Randall A. Lewis & Justin M. Rao, 2015. "The Unfavorable Economics of Measuring the Returns to Advertising," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 130(4), pages 1941-1973.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Patrick Bajari & Brian Burdick & Guido W. Imbens & Lorenzo Masoero & James McQueen & Thomas Richardson & Ido M. Rosen, 2021. "Multiple Randomization Designs," Papers 2112.13495, arXiv.org.
    2. Jonathan Roth & Pedro H. C. Sant’Anna, 2023. "Efficient Estimation for Staggered Rollout Designs," Journal of Political Economy Microeconomics, University of Chicago Press, vol. 1(4), pages 669-709.
    3. Athey, Susan & Imbens, Guido W., 2022. "Design-based analysis in Difference-In-Differences settings with staggered adoption," Journal of Econometrics, Elsevier, vol. 226(1), pages 62-79.
    4. Han, Kevin & Basse, Guillaume & Bojinov, Iavor, 2024. "Population interference in panel experiments," Journal of Econometrics, Elsevier, vol. 238(1).
    5. Maria Petrova & Ananya Sen & Pinar Yildirim, 2021. "Social Media and Political Contributions: The Impact of New Technology on Political Competition," Management Science, INFORMS, vol. 67(5), pages 2997-3021, May.
    6. Shuze Chen & David Simchi-Levi & Chonghuan Wang, 2024. "Experimenting on Markov Decision Processes with Local Treatments," Papers 2407.19618, arXiv.org, revised Oct 2024.
    7. Retsef Levi & Elisabeth Paulson & Georgia Perakis & Emily Zhang, 2024. "Heterogeneous Treatment Effects in Panel Data," Papers 2406.05633, arXiv.org.
    8. Jinglong Zhao, 2023. "Adaptive Neyman Allocation," Papers 2309.08808, arXiv.org, revised Sep 2023.
    9. Kathryn N. Vasilaky & J. Michelle Brock, 2020. "Power(ful) guidelines for experimental economists," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 6(2), pages 189-212, December.
    10. Jinglong Zhao & Zijie Zhou, 2022. "Pigeonhole Design: Balancing Sequential Experiments from an Online Matching Perspective," Papers 2201.12936, arXiv.org, revised May 2024.
    11. Iavor Bojinov & David Simchi-Levi & Jinglong Zhao, 2023. "Design and Analysis of Switchback Experiments," Management Science, INFORMS, vol. 69(7), pages 3759-3777, July.
    12. Jinglong Zhao, 2024. "Experimental Design For Causal Inference Through An Optimization Lens," Papers 2408.09607, arXiv.org, revised Aug 2024.
    13. Vivek F. Farias & Andrew A. Li & Tianyi Peng, 2021. "Learning Treatment Effects in Panels with General Intervention Patterns," Papers 2106.02780, arXiv.org, revised Mar 2023.
    14. Shan Huang & Chen Wang & Yuan Yuan & Jinglong Zhao & Jingjing Zhang, 2023. "Estimating Effects of Long-Term Treatments," Papers 2308.08152, arXiv.org.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Xiong, Ruoxuan & Pelger, Markus, 2023. "Large dimensional latent factor modeling with missing observations and applications to causal inference," Journal of Econometrics, Elsevier, vol. 233(1), pages 271-301.
    2. Dmitry Arkhangelsky & Guido Imbens, 2023. "Causal Models for Longitudinal and Panel Data: A Survey," Papers 2311.15458, arXiv.org, revised Jun 2024.
    3. Bruno Ferman & Cristine Pinto, 2021. "Synthetic controls with imperfect pretreatment fit," Quantitative Economics, Econometric Society, vol. 12(4), pages 1197-1221, November.
    4. Susan Athey & Mohsen Bayati & Nikolay Doudchenko & Guido Imbens & Khashayar Khosravi, 2021. "Matrix Completion Methods for Causal Panel Data Models," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 116(536), pages 1716-1730, October.
    5. Dennis Shen & Peng Ding & Jasjeet Sekhon & Bin Yu, 2022. "Same Root Different Leaves: Time Series and Cross-Sectional Methods in Panel Data," Papers 2207.14481, arXiv.org, revised Oct 2022.
    6. Alberto Abadie & Anish Agarwal & Raaz Dwivedi & Abhin Shah, 2024. "Doubly Robust Inference in Causal Latent Factor Models," Papers 2402.11652, arXiv.org, revised Oct 2024.
    7. Lea Bottmer & Guido Imbens & Jann Spiess & Merrill Warnick, 2021. "A Design-Based Perspective on Synthetic Control Methods," Papers 2101.09398, arXiv.org, revised Jul 2023.
    8. Davide Viviano & Jelena Bradic, 2021. "Dynamic covariate balancing: estimating treatment effects over time with potential local projections," Papers 2103.01280, arXiv.org, revised Jan 2024.
    9. Dmitry Arkhangelsky & Susan Athey & David A. Hirshberg & Guido W. Imbens & Stefan Wager, 2021. "Synthetic Difference-in-Differences," American Economic Review, American Economic Association, vol. 111(12), pages 4088-4118, December.
    10. Viviano, Davide & Bradic, Jelena, 2023. "Synthetic Learner: Model-free inference on treatments over time," Journal of Econometrics, Elsevier, vol. 234(2), pages 691-713.
    11. Li, Xingyu & Shen, Yan & Zhou, Qiankun, 2024. "Confidence intervals of treatment effects in panel data models with interactive fixed effects," Journal of Econometrics, Elsevier, vol. 240(1).
    12. Davide Viviano & Jelena Bradic, 2019. "Synthetic learner: model-free inference on treatments over time," Papers 1904.01490, arXiv.org, revised Aug 2022.
    13. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    14. Jason Poulos & Andrea Albanese & Andrea Mercatanti & Fan Li, 2021. "Retrospective causal inference via matrix completion, with an evaluation of the effect of European integration on cross-border employment," Papers 2106.00788, arXiv.org.
    15. Bai, Jushan & Wang, Peng, 2024. "Causal inference using factor models," MPRA Paper 120585, University Library of Munich, Germany.
    16. Victor Chernozhukov & Kaspar Wüthrich & Yinchu Zhu, 2021. "An Exact and Robust Conformal Inference Method for Counterfactual and Synthetic Controls," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 116(536), pages 1849-1864, October.
    17. Eli Ben‐Michael & Avi Feller & Jesse Rothstein, 2022. "Synthetic controls with staggered adoption," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 84(2), pages 351-381, April.
    18. Dallas Dotter & Duncan Chaplin & Maria Bartlett, "undated". "Impacts of School Reforms in Washington, DC on Student Achievement," Mathematica Policy Research Reports 44e95d7566434a21b8d57f951, Mathematica Policy Research.
    19. Roth, Jonathan & Sant’Anna, Pedro H.C. & Bilinski, Alyssa & Poe, John, 2023. "What’s trending in difference-in-differences? A synthesis of the recent econometrics literature," Journal of Econometrics, Elsevier, vol. 235(2), pages 2218-2244.
    20. Florian Gunsilius, 2020. "Distributional synthetic controls," Papers 2001.06118, arXiv.org, revised Dec 2021.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ecl:stabus:3837. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/gsstaus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.