[go: up one dir, main page]
IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2002.05819.html
   My bibliography  Save this paper

Fairness through Experimentation: Inequality in A/B testing as an approach to responsible design

Author

Listed:
  • Guillaume Saint-Jacques
  • Amir Sepehri
  • Nicole Li
  • Igor Perisic
Abstract
As technology continues to advance, there is increasing concern about individuals being left behind. Many businesses are striving to adopt responsible design practices and avoid any unintended consequences of their products and services, ranging from privacy vulnerabilities to algorithmic bias. We propose a novel approach to fairness and inclusiveness based on experimentation. We use experimentation because we want to assess not only the intrinsic properties of products and algorithms but also their impact on people. We do this by introducing an inequality approach to A/B testing, leveraging the Atkinson index from the economics literature. We show how to perform causal inference over this inequality measure. We also introduce the concept of site-wide inequality impact, which captures the inclusiveness impact of targeting specific subpopulations for experiments, and show how to conduct statistical inference on this impact. We provide real examples from LinkedIn, as well as an open-source, highly scalable implementation of the computation of the Atkinson index and its variance in Spark/Scala. We also provide over a year's worth of learnings -- gathered by deploying our method at scale and analyzing thousands of experiments -- on which areas and which kinds of product innovations seem to inherently foster fairness through inclusiveness.

Suggested Citation

  • Guillaume Saint-Jacques & Amir Sepehri & Nicole Li & Igor Perisic, 2020. "Fairness through Experimentation: Inequality in A/B testing as an approach to responsible design," Papers 2002.05819, arXiv.org.
    • Handle: RePEc:arx:papers:2002.05819
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2002.05819
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. John Creedy, 2016. "Interpreting inequality measures and changes in inequality," New Zealand Economic Papers, Taylor & Francis Journals, vol. 50(2), pages 177-192, August.
    2. Das, T & Parikh, A, 1982. "Decomposition of Inequality Measures and a Comparative Analysis," Empirical Economics, Springer, vol. 7(1-2), pages 23-48.
    3. Anthony B. Atkinson & Andrea Brandolini, 2010. "On Analyzing the World Distribution of Income," The World Bank Economic Review, World Bank, vol. 24(1), pages 1-37, January.
    4. Atkinson, Anthony B., 1970. "On the measurement of inequality," Journal of Economic Theory, Elsevier, vol. 2(3), pages 244-263, September.
    5. repec:ebl:ecbull:v:4:y:2003:i:29:p:1-12 is not listed on IDEAS
    6. Mª Casilda Lasso de la Vega & Ana Marta Urrutia, 2003. "A new factorial decomposition for the atkinson measure," Economics Bulletin, AccessEcon, vol. 4(29), pages 1-12.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Sumie Sato & Mototsugu Fukushige, 2012. "The Chinese economy and income inequality among East Asian countries," China Economic Journal, Taylor & Francis Journals, vol. 5(1), pages 1-15.
    2. Miguel Niño‐Zarazúa & Laurence Roope & Finn Tarp, 2017. "Global Inequality: Relatively Lower, Absolutely Higher," Review of Income and Wealth, International Association for Research in Income and Wealth, vol. 63(4), pages 661-684, December.
    3. Casilda Lasso de la Vega & Ana Urrutia, 2008. "The ‘Extended’ Atkinson family: The class of multiplicatively decomposable inequality measures, and some new graphical procedures for analysts," The Journal of Economic Inequality, Springer;Society for the Study of Economic Inequality, vol. 6(2), pages 211-225, June.
    4. Laurence Roope, 2015. "Critical percentiles for equalizing growth," CSAE Working Paper Series 2015-16, Centre for the Study of African Economies, University of Oxford.
    5. Christoph Lakner & Branko Milanovic, 2016. "Global Income Distribution: From the Fall of the Berlin Wall to the Great Recession," The World Bank Economic Review, World Bank, vol. 30(2), pages 203-232.
    6. Rolf Aaberge & François Bourguignon & Andrea Brandolini & Francisco H. G. Ferreira & Janet C. Gornick & John Hills & Markus Jäntti & Stephen P. Jenkins & Eric Marlier & John Micklewright & Brian Nolan, 2017. "Tony Atkinson and his Legacy," Review of Income and Wealth, International Association for Research in Income and Wealth, vol. 63(3), pages 411-444, September.
    7. Pedro Conceição & Yanchun Zhang, 2010. "Discounting in the context of climate change economics: the policy implications of uncertainty and global asymmetries," Environmental Economics and Policy Studies, Springer;Society for Environmental Economics and Policy Studies - SEEPS, vol. 12(1), pages 31-57, June.
    8. Jasso, Guillermina & Kotz, Samuel, 2007. "Two Types of Inequality: Inequality Between Persons and Inequality Between Subgroups," IZA Discussion Papers 2749, Institute of Labor Economics (IZA).
    9. Rolf Aaberge & Andrea Brandolini, 2014. "Multidimensional poverty and inequality," Temi di discussione (Economic working papers) 976, Bank of Italy, Economic Research and International Relations Area.
    10. Paolo Li Donni & Vito Peragine & Giuseppe Pignataro, 2014. "Ex‐Ante And Ex‐Post Measurement Of Equality Of Opportunity In Health: A Normative Decomposition," Health Economics, John Wiley & Sons, Ltd., vol. 23(2), pages 182-198, February.
    11. Buchholz, Wolfgang & Schumacher, Jan, 2010. "Discounting and welfare analysis over time: Choosing the [eta]," European Journal of Political Economy, Elsevier, vol. 26(3), pages 372-385, September.
    12. James E. Foster & Joel Greer & Erik Thorbecke, 2010. "The Foster-Greer-Thorbecke (FGT) Poverty Measures: Twenty-Five Years Later," Working Papers 2010-14, The George Washington University, Institute for International Economic Policy.
    13. Flaviana Palmisano & Dirk Van de gaer, 2016. "History-dependent growth incidence: a characterization and an application to the economic crisis in Italy," Oxford Economic Papers, Oxford University Press, vol. 68(2), pages 585-603.
    14. Stéphane Mussard & Françoise Seyte & Michel Terraza, 2006. "La décomposition de l’indicateur de Gini en sous-groupes : une revue de la littérature," Cahiers de recherche 06-11, Departement d'économique de l'École de gestion à l'Université de Sherbrooke.
    15. La-Bhus Fah Jirasavetakul & Christoph Lakner, 2020. "The Distribution of Consumption Expenditure in Sub-Saharan Africa: The Inequality Among All Africans," Journal of African Economies, Centre for the Study of African Economies, vol. 29(1), pages 1-25.
    16. Carla Canelas & Rachel M. Gisselquist, 2019. "Horizontal Inequality and Data Challenges," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 143(1), pages 157-172, May.
    17. Daniel Suryadarma & Rima Prama Artha & Asep Suryahadi & Sudarno Sumarto, . "A Reassessment of Inequality and Its Role in Poverty Reduction in Indonesia," Journal Article, Publications Department.
    18. Creedy, John, 2019. "The Atkinson Inequality Measure and Inequality Aversion," Working Paper Series 20918, Victoria University of Wellington, Chair in Public Finance.
    19. Glenn Sheriff & Kelly B. Maguire, 2013. "Ranking Distributions of Environmental Outcomes Across Population Groups," NCEE Working Paper Series 201304, National Center for Environmental Economics, U.S. Environmental Protection Agency, revised Aug 2013.
    20. Dietz, Simon & Hepburn, Cameron, 2013. "Benefit–cost analysis of non-marginal climate and energy projects," Energy Economics, Elsevier, vol. 40(C), pages 61-71.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2002.05819. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.