[go: up one dir, main page]
IDEAS home Printed from https://ideas.repec.org/a/taf/gnstxx/v31y2019i1p100-130.html
   My bibliography  Save this article

Using the area under an estimated ROC curve to test the adequacy of binary predictors

Author

Listed:
  • Robert P. Lieli
  • Yu-Chin Hsu
Abstract
We consider using the area under an empirical receiver operating characteristic curve to test the hypothesis that a predictive index combined with a range of cutoffs performs no better than pure chance in forecasting a binary outcome. This corresponds to the null hypothesis that the area in question, denoted as AUC, is 1/2. We show that if the predictive index comes from a first-stage regression model estimated over the same data set, then testing the null based on the standard asymptotic normality results leads to severe size distortion in general settings. We then analytically derive the proper asymptotic null distribution of the empirical AUC in a special case; namely, when the first-stage regressors are Bernoulli random variables. This distribution can be utilised to construct a fully in-sample test of $ H_0: {\rm AUC}=1/2 $ H0:AUC=1/2 with correct size and more power than out-of-sample tests based on sample splitting, though practical application becomes cumbersome with more than two regressors.

Suggested Citation

  • Robert P. Lieli & Yu-Chin Hsu, 2019. "Using the area under an estimated ROC curve to test the adequacy of binary predictors," Journal of Nonparametric Statistics, Taylor & Francis Journals, vol. 31(1), pages 100-130, January.
    • Handle: RePEc:taf:gnstxx:v:31:y:2019:i:1:p:100-130
    DOI: 10.1080/10485252.2018.1537440
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1080/10485252.2018.1537440
    Download Restriction: Access to full text is restricted to subscribers.
    File URL: https://libkey.io/10.1080/10485252.2018.1537440?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to look for a different version below or search for a different version of it.

    Other versions of this item:

    References listed on IDEAS

    as
    1. Lieli, Robert P. & White, Halbert, 2010. "The construction of empirical credit scoring rules based on maximization principles," Journal of Econometrics, Elsevier, vol. 157(1), pages 110-119, July.
    2. Robert P. Lieli & Yu-Chin Hsu, 2019. "Using the area under an estimated ROC curve to test the adequacy of binary predictors," Journal of Nonparametric Statistics, Taylor & Francis Journals, vol. 31(1), pages 100-130, January.
    3. Elliott, Graham & Lieli, Robert P., 2013. "Predicting binary outcomes," Journal of Econometrics, Elsevier, vol. 174(1), pages 15-26.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Robert P. Lieli & Yu-Chin Hsu, 2019. "Using the area under an estimated ROC curve to test the adequacy of binary predictors," Journal of Nonparametric Statistics, Taylor & Francis Journals, vol. 31(1), pages 100-130, January.
    2. Halko, Marja-Liisa & Lappalainen, Olli & Sääksvuori, Lauri, 2021. "Do non-choice data reveal economic preferences? Evidence from biometric data and compensation-scheme choice," Journal of Economic Behavior & Organization, Elsevier, vol. 188(C), pages 87-104.
    3. Miguel Angel Saldarriaga, 2017. "Credit Booms in Commodity Exporters," Working Papers 98, Peruvian Economic Association.
    4. Christiansen, Charlotte & Eriksen, Jonas N. & Møller, Stig V., 2019. "Negative house price co-movements and US recessions," Regional Science and Urban Economics, Elsevier, vol. 77(C), pages 382-394.
    5. Kajal Lahiri & Cheng Yang, 2023. "ROC and PRC Approaches to Evaluate Recession Forecasts," Journal of Business Cycle Research, Springer;Centre for International Research on Economic Tendency Surveys (CIRET), vol. 19(2), pages 119-148, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chen, Le-Yu & Lee, Sokbae, 2018. "Best subset binary prediction," Journal of Econometrics, Elsevier, vol. 206(1), pages 39-56.
    2. Kajal Lahiri & Cheng Yang, 2023. "ROC and PRC Approaches to Evaluate Recession Forecasts," Journal of Business Cycle Research, Springer;Centre for International Research on Economic Tendency Surveys (CIRET), vol. 19(2), pages 119-148, September.
    3. Halbert White & Karim Chalak, 2008. "Identifying Structural Effects in Nonseparable Systems Using Covariates," Boston College Working Papers in Economics 734, Boston College Department of Economics.
    4. Toru Kitagawa & Aleksey Tetenov, 2018. "Who Should Be Treated? Empirical Welfare Maximization Methods for Treatment Choice," Econometrica, Econometric Society, vol. 86(2), pages 591-616, March.
    5. Baidoo, Edwin & Natarajan, Ramachandran, 2021. "Profit-based credit models with lender’s attitude towards risk and loss," Journal of Behavioral and Experimental Finance, Elsevier, vol. 32(C).
    6. Toru Kitagawa & Aleksey Tetenov, 2015. "Who should be treated? Empirical welfare maximization methods for treatment choice," CeMMAP working papers 10/15, Institute for Fiscal Studies.
    7. Lahiri, Kajal & Yang, Liu, 2013. "Forecasting Binary Outcomes," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1025-1106, Elsevier.
    8. Graham Elliott & Allan Timmermann, 2016. "Economic Forecasting," Economics Books, Princeton University Press, edition 1, number 10740.
    9. Martin Feldkircher & Thomas Gruber & Isabella Moder, 2014. "Using a Threshold Approach to Flag Vulnerabilities in CESEE Economies," Focus on European Economic Integration, Oesterreichische Nationalbank (Austrian Central Bank), issue 3, pages 8-30.
    10. Blaskowitz, Oliver & Herwartz, Helmut, 2011. "On economic evaluation of directional forecasts," International Journal of Forecasting, Elsevier, vol. 27(4), pages 1058-1065, October.
    11. Florios, Kostas & Skouras, Spyros, 2008. "Exact computation of max weighted score estimators," Journal of Econometrics, Elsevier, vol. 146(1), pages 86-91, September.
    12. Christiansen, Charlotte & Eriksen, Jonas N. & Møller, Stig V., 2019. "Negative house price co-movements and US recessions," Regional Science and Urban Economics, Elsevier, vol. 77(C), pages 382-394.
    13. Ghysels, Eric & Babii, Andrii & Chen, Xi & Kumar, Rohit, 2020. "Binary Choice with Asymmetric Loss in a Data-Rich Environment: Theory and an Application to Racial Justice," CEPR Discussion Papers 15418, C.E.P.R. Discussion Papers.
    14. André K. Anundsen & Karsten Gerdrup & Frank Hansen & Kasper Kragh‐Sørensen, 2016. "Bubbles and Crises: The Role of House Prices and Credit," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 31(7), pages 1291-1311, November.
    15. Rafał Balina & Marta Idasz-Balina, 2021. "Drivers of Individual Credit Risk of Retail Customers—A Case Study on the Example of the Polish Cooperative Banking Sector," Risks, MDPI, vol. 9(12), pages 1-26, December.
    16. Stanislav Anatolyev & Natalia Kryzhanovskaya, 2009. "Directional Prediction of Returns under Asymmetric Loss: Direct and Indirect Approaches," Working Papers w0136, New Economic School (NES).
    17. Davide Viviano & Jess Rudder, 2020. "Policy design in experiments with unknown interference," Papers 2011.08174, arXiv.org, revised May 2024.
    18. Drehmann, Mathias & Juselius, Mikael, 2014. "Evaluating early warning indicators of banking crises: Satisfying policy requirements," International Journal of Forecasting, Elsevier, vol. 30(3), pages 759-780.
    19. Nyberg, Henri, 2011. "Forecasting the direction of the US stock market with dynamic binary probit models," International Journal of Forecasting, Elsevier, vol. 27(2), pages 561-578, April.
    20. Lee, Seung Jung & Posenau, Kelly E. & Stebunovs, Viktors, 2020. "The anatomy of financial vulnerabilities and banking crises," Journal of Banking & Finance, Elsevier, vol. 112(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:taf:gnstxx:v:31:y:2019:i:1:p:100-130. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Longhurst (email available below). General contact details of provider: http://www.tandfonline.com/GNST20 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.