[go: up one dir, main page]
IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2212.05866.html
   My bibliography  Save this paper

Measuring the Driving Forces of Predictive Performance: Application to Credit Scoring

Author

Listed:
  • Hu'e Sullivan
  • Hurlin Christophe
  • P'erignon Christophe
  • Saurin S'ebastien
Abstract
In credit scoring, machine learning models are known to outperform standard parametric models. As they condition access to credit, banking supervisors and internal model validation teams need to monitor their predictive performance and to identify the features with the highest impact on performance. To facilitate this, we introduce the XPER methodology to decompose a performance metric (e.g., AUC, $R^2$) into specific contributions associated with the various features of a classification or regression model. XPER is theoretically grounded on Shapley values and is both model-agnostic and performance metric-agnostic. Furthermore, it can be implemented either at the model level or at the individual level. Using a novel dataset of car loans, we decompose the AUC of a machine-learning model trained to forecast the default probability of loan applicants. We show that a small number of features can explain a surprisingly large part of the model performance. Furthermore, we find that the features that contribute the most to the predictive performance of the model may not be the ones that contribute the most to individual forecasts (SHAP). We also show how XPER can be used to deal with heterogeneity issues and significantly boost out-of-sample performance.

Suggested Citation

  • Hu'e Sullivan & Hurlin Christophe & P'erignon Christophe & Saurin S'ebastien, 2022. "Measuring the Driving Forces of Predictive Performance: Application to Credit Scoring," Papers 2212.05866, arXiv.org, revised Jun 2023.
    • Handle: RePEc:arx:papers:2212.05866
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2212.05866
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Shorrocks, A F, 1980. "The Class of Additively Decomposable Inequality Measures," Econometrica, Econometric Society, vol. 48(3), pages 613-625, April.
    2. Bourguignon, Francois, 1979. "Decomposable Income Inequality Measures," Econometrica, Econometric Society, vol. 47(4), pages 901-920, July.
    3. Mukund Sundararajan & Amir Najmi, 2019. "The many Shapley values for model explanation," Papers 1908.08474, arXiv.org, revised Feb 2020.
    4. Gunnarsson, Björn Rafn & vanden Broucke, Seppe & Baesens, Bart & Óskarsdóttir, María & Lemahieu, Wilfried, 2021. "Deep learning for credit scoring: Do or don’t?," European Journal of Operational Research, Elsevier, vol. 295(1), pages 292-305.
    5. Daniel W. Apley & Jingyu Zhu, 2020. "Visualizing the effects of predictor variables in black box supervised learning models," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 82(4), pages 1059-1086, September.
    6. Lessmann, Stefan & Baesens, Bart & Seow, Hsin-Vonn & Thomas, Lyn C., 2015. "Benchmarking state-of-the-art classification algorithms for credit scoring: An update of research," European Journal of Operational Research, Elsevier, vol. 247(1), pages 124-136.
    7. Bart Baesens & Rudy Setiono & Christophe Mues & Jan Vanthienen, 2003. "Using Neural Network Rule Extraction and Decision Tables for Credit-Risk Evaluation," Management Science, INFORMS, vol. 49(3), pages 312-329, March.
    8. Frédéric Chantreuil & Sébastien Courtin & Kevin Fourrey & Isabelle Lebon, 2019. "A note on the decomposability of inequality measures," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 53(2), pages 283-298, August.
    9. Shorrocks, Anthony F, 1984. "Inequality Decomposition by Population Subgroups," Econometrica, Econometric Society, vol. 52(6), pages 1369-1385, November.
    10. Osnat Israeli, 2007. "A Shapley-based decomposition of the R-Square of a linear regression," The Journal of Economic Inequality, Springer;Society for the Study of Economic Inequality, vol. 5(2), pages 199-212, August.
    11. Shorrocks, A F, 1982. "Inequality Decomposition by Factor Components," Econometrica, Econometric Society, vol. 50(1), pages 193-211, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chantreuil, Frédéric & Fourrey, Kévin & Lebon, Isabelle & Rebière, Thérèse, 2021. "Magnitude and evolution of gender and race contributions to earnings inequality across US regions," Research in Economics, Elsevier, vol. 75(1), pages 45-59.
    2. Chantreuil, Frédéric & Fourrey, Kévin & Lebon, Isabelle & Rebiere, Therese, 2020. "Decomposing US Income Inequality à La Shapley: Race Matters, but Gender Too," IZA Discussion Papers 12950, Institute of Labor Economics (IZA).
    3. Teixidó Figueras, Jordi & Duro Moreno, Juan Antonio, 2012. "Ecological Footprint Inequality: A methodological review and some results," Working Papers 2072/203168, Universitat Rovira i Virgili, Department of Economics.
    4. Arthur Charpentier & Stéphane Mussard, 2011. "Income inequality games," The Journal of Economic Inequality, Springer;Society for the Study of Economic Inequality, vol. 9(4), pages 529-554, December.
    5. Mussard, Stéphane & Pi Alperin, Maria Noel, 2011. "Poverty growth in Scandinavian countries: A Sen multi-decomposition," Economic Modelling, Elsevier, vol. 28(6), pages 2842-2853.
    6. F. Chantreuil & A. Trannoy, 1999. "Inequality decomposition values : the trade-off between marginality and consistency," THEMA Working Papers 99-24, THEMA (THéorie Economique, Modélisation et Applications), Université de Cergy-Pontoise.
    7. Francesca Battisti & Francesco Porro, 2023. "A multi-decomposition of Zenga-84 inequality index: an application to the disparity in CO $$_2$$ 2 emissions in European countries," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 32(3), pages 957-981, September.
    8. Brulhart, Marius & Traeger, Rolf, 2005. "An account of geographic concentration patterns in Europe," Regional Science and Urban Economics, Elsevier, vol. 35(6), pages 597-624, November.
    9. Guanghua Wan & Zhangyue Zhou, 2005. "Income Inequality in Rural China: Regression‐based Decomposition Using Household Data," Review of Development Economics, Wiley Blackwell, vol. 9(1), pages 107-120, February.
    10. Stéphane Mussard & Michel Terraza, 2009. "Décompositions des mesures d'inégalité : le cas des coefficients de Gini et d'entropie," Recherches économiques de Louvain, De Boeck Université, vol. 75(2), pages 151-181.
    11. Guido Erreygers & Roselinde Kessels, 2013. "Regression-Based Decompositions of Rank-Dependent Indicators of Socioeconomic Inequality of Health," Research on Economic Inequality, in: Health and Inequality, volume 21, pages 227-259, Emerald Group Publishing Limited.
    12. Anthony Shorrocks, 2013. "Decomposition procedures for distributional analysis: a unified framework based on the Shapley value," The Journal of Economic Inequality, Springer;Society for the Study of Economic Inequality, vol. 11(1), pages 99-126, March.
    13. Remuzgo, Lorena & Sarabia, José María, 2013. "Desigualdad en la distribución mundial de emisiones de CO2 por sectores: Descomposición y estudio de sensibilidad/Inequality of Global Distribution of CO2 Emissions by Sector: Decomposition and Sensit," Estudios de Economia Aplicada, Estudios de Economia Aplicada, vol. 31, pages 65-92, Enero.
    14. Sebastian Leitner, 2015. "Drivers of wealth inequality in euro area countries," Working Paper Reihe der AK Wien - Materialien zu Wirtschaft und Gesellschaft 137, Kammer für Arbeiter und Angestellte für Wien, Abteilung Wirtschaftswissenschaft und Statistik.
    15. Muszyńska Joanna & Wędrowska Ewa, 2018. "Income Inequality of Households in Poland: A Subgroup Decomposition of Generalized Entropy Measures," Econometrics. Advances in Applied Data Analysis, Sciendo, vol. 22(4), pages 43-64, December.
    16. Lwin Lwin Aung & Peter Warr, 2021. "Decomposing changes in inequality: Evidence from Myanmar," Review of Development Economics, Wiley Blackwell, vol. 25(3), pages 1172-1196, August.
    17. C. Chameni Nembua, 2008. "The 'natural' bidimensional decomposition of inequality indices: evaluating factor contributions to households welfare inequality in Cameroon, 1996-2001," Applied Economics Letters, Taylor & Francis Journals, vol. 15(12), pages 963-970.
    18. Frank Cowell & Carlo Fiorio, 2011. "Inequality decompositions—a reconciliation," The Journal of Economic Inequality, Springer;Society for the Study of Economic Inequality, vol. 9(4), pages 509-528, December.
    19. Chen, Yujia & Calabrese, Raffaella & Martin-Barragan, Belen, 2024. "Interpretable machine learning for imbalanced credit scoring datasets," European Journal of Operational Research, Elsevier, vol. 312(1), pages 357-372.
    20. Muszyńska Joanna & Oczki Jarosław & Wędrowska Ewa, 2018. "Income Inequality in Poland and the United Kingdom. Decomposition of the Theil Index," Folia Oeconomica Stetinensia, Sciendo, vol. 18(1), pages 108-122, June.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2212.05866. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.