[go: up one dir, main page]
IDEAS home Printed from https://ideas.repec.org/p/mia/wpaper/2016-08.html
   My bibliography  Save this paper

Accounting for Multiplicity in Inference on Economics Journal Rankings

Author

Listed:
  • William Horrace

    (Syracuse University)

  • Christopher Parmeter

    (University of Miami)

Abstract
Nearly all journal rankings are constructed as deterministic, yet they are clearly stochastic. To remedy this, Stern (2013) calculates standard errors and performs inference on the ranks of economics journal based on impact factors. However, this inference is essentially a series of univariate tests that do not control for the overall error rate of the inferential exercise. Using multiple comparison and ranking procedures, we reevaluate the inference while controlling for the multiplicity (the implied multivariate inference) in the rank statistic. The results are compared and di_erences highlighted.

Suggested Citation

  • William Horrace & Christopher Parmeter, 2016. "Accounting for Multiplicity in Inference on Economics Journal Rankings," Working Papers 2016-08, University of Miami, Department of Economics.
    • Handle: RePEc:mia:wpaper:2016-08
    as

    Download full text from publisher

    File URL: https://www.herbert.miami.edu/_assets/files/repec/WP2016-08.pdf
    File Function: First version, 2016
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Chia-Lin Chang & Michael Mcaleer, 2014. "Just How Good Are The Top Three Journals In Finance? An Assessment Based On Quantity And Quality Citations," Annals of Financial Economics (AFE), World Scientific Publishing Co. Pte. Ltd., vol. 9(01), pages 1-31.
    2. William Horrace & Seth Richards-Shubik & Ian Wright, 2015. "Expected efficiency ranks from parametric stochastic frontier models," Empirical Economics, Springer, vol. 48(2), pages 829-848, March.
    3. Andrew I. Friedson & William C. Horrace & Allison F. Marier, 2019. "So Many Hospitals, So Little Information: How Hospital Value‐Based Purchasing Is a Game of Chance," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 773-799, October.
    4. Chia-Lin Chang & Michael McAleer, 2013. "Ranking Leading Econometrics Journals Using Citations Data from ISI and RePEc," Econometrics, MDPI, vol. 1(3), pages 1-19, November.
    5. William C. Horrace & Peter Schmidt, 2000. "Multiple comparisons with the best, with economic applications," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 15(1), pages 1-26.
    6. William Horrace & Joseph Marchand & Timothy Smeeding, 2008. "Ranking inequality: Applications of multivariate subset selection," The Journal of Economic Inequality, Springer;Society for the Study of Economic Inequality, vol. 6(1), pages 5-32, March.
    7. Ignacio Palacios-Huerta & Oscar Volij, 2004. "The Measurement of Intellectual Influence," Econometrica, Econometric Society, vol. 72(3), pages 963-977, May.
    8. David I. Stern, 2013. "Uncertainty Measures for Economics Journal Impact Factors," Journal of Economic Literature, American Economic Association, vol. 51(1), pages 173-189, March.
    9. Chia-Lin Chang & Esfandiar Maasoumi & Michael McAleer, 2016. "Robust Ranking of Journal Quality: An Application to Economics," Econometric Reviews, Taylor & Francis Journals, vol. 35(1), pages 50-97, January.
    10. Chia-Lin Chang & Michael Mcaleer, 2013. "What Do Experts Know About Forecasting Journal Quality? A Comparison With Isi Research Impact In Finance," Annals of Financial Economics (AFE), World Scientific Publishing Co. Pte. Ltd., vol. 8(01), pages 1-30.
    11. William C. Horrace, 2005. "On the ranking uncertainty of labor market wage gaps," Journal of Population Economics, Springer;European Society for Population Economics, vol. 18(1), pages 181-187, September.
    12. David Card & Stefano DellaVigna, 2013. "Nine Facts about Top Journals in Economics," Journal of Economic Literature, American Economic Association, vol. 51(1), pages 144-161, March.
    13. Pierre-Philippe Combes & Laurent Linnemer, 2010. "Inferring Missing Citations: A Quantitative Multi-Criteria Ranking of all Journals in Economics," Working Papers halshs-00520325, HAL.
    14. Kalaitzidakis, P. & Mamuneas, T.P. & Stengos, T., 2003. "Rankings of Academic Journals and Institutions," Working Papers 2003-8, University of Guelph, Department of Economics and Finance.
    15. Baltagi, Badi H., 2007. "Worldwide Econometrics Rankings: 1989–2005," Econometric Theory, Cambridge University Press, vol. 23(5), pages 952-1012, October.
    16. Pantelis Kalaitzidakis & Theofanis P. Mamuneas & Thanasis Stengos, 2003. "Rankings of Academic Journals and Institutions in Economics," Journal of the European Economic Association, MIT Press, vol. 1(6), pages 1346-1366, December.
    17. Hong Li & Wei Ning, 2012. "Multiple comparisons with a control under heteroscedasticity," Journal of Applied Statistics, Taylor & Francis Journals, vol. 39(10), pages 2275-2283, June.
    18. Alfonso Flores-Lagunes & William C. Horrace & Kurt E. Schnier, 2007. "Identifying technically efficient fishing vessels: a non-empty, minimal subset approach," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 22(4), pages 729-745.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Andrew I. Friedson & William C. Horrace & Allison F. Marier, 2019. "So Many Hospitals, So Little Information: How Hospital Value‐Based Purchasing Is a Game of Chance," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 773-799, October.
    2. Johannes König & David I. Stern & Richard S.J. Tol, 2022. "Confidence Intervals for Recursive Journal Impact Factors," Tinbergen Institute Discussion Papers 22-038/III, Tinbergen Institute.
    3. William C. Horrace & Hyunseok Jung & Jonathan L. Pressler & Amy Ellen Schwartz, 2021. "What Makes a Classmate a Peer? Examining Which Peers Matter in NYC Elementary Schools," Center for Policy Research Working Papers 241, Center for Policy Research, Maxwell School, Syracuse University.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lutz Bornmann & Alexander Butz & Klaus Wohlrabe, 2018. "What are the top five journals in economics? A new meta-ranking," Applied Economics, Taylor & Francis Journals, vol. 50(6), pages 659-675, February.
    2. Wohlrabe, Klaus, 2016. "Taking the Temperature: A Meta-Ranking of Economics Journals," MPRA Paper 68933, University Library of Munich, Germany.
    3. David L. Anderson & John Tressler, 2017. "Researcher rank stability across alternative output measurement schemes in the context of a time limited research evaluation: the New Zealand case," Applied Economics, Taylor & Francis Journals, vol. 49(45), pages 4542-4553, September.
    4. Önder Ali Sina & Yilmazkuday Hakan, 2020. "Thirty-Five Years of Peer-Reviewed Publishing by North American Economics PhDs: Quantity, Quality, and Beyond," Open Economics, De Gruyter, vol. 3(1), pages 70-85, January.
    5. Ali Sina Önder & Sascha Schweitzer & Hakan Yilmazkuday, 2021. "Field Distance and Quality in Economists’ Collaborations," Working Papers in Economics & Finance 2021-04, University of Portsmouth, Portsmouth Business School, Economics and Finance Subject Group.
    6. J. Atsu Amegashie, 2020. "Citations And Incentives In Academic Contests," Economic Inquiry, Western Economic Association International, vol. 58(3), pages 1233-1244, July.
    7. Johannes König & David I. Stern & Richard S.J. Tol, 2022. "Confidence Intervals for Recursive Journal Impact Factors," Tinbergen Institute Discussion Papers 22-038/III, Tinbergen Institute.
    8. Chagas , André Luis Squarize, 2017. "Publish or Perish: um ranking de revistas da subárea de Economia Regional e Urbana para os pesquisadores brasileiros," Revista Brasileira de Estudos Regionais e Urbanos, Associação Brasileira de Estudos Regionais e Urbanos (ABER), vol. 11(4), pages 515-536.
    9. Önder, Ali Sina & Schweitzer, Sascha & Yilmazkuday, Hakan, 2021. "Specialization, field distance, and quality in economists’ collaborations," Journal of Informetrics, Elsevier, vol. 15(4).
    10. James J. Heckman & Sidharth Moktan, 2020. "Publishing and Promotion in Economics: The Tyranny of the Top Five," Journal of Economic Literature, American Economic Association, vol. 58(2), pages 419-470, June.
    11. Lutz Bornmann & Klaus Wohlrabe, 2019. "Normalisation of citation impact in economics," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(2), pages 841-884, August.
    12. Ali Sina Önder & Sascha Schweitzer, 2017. "Catching up or falling behind? Promising changes and persistent patterns across cohorts of economics PhDs in German-speaking countries from 1991 to 2008," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(3), pages 1297-1331, March.
    13. Kosnik, Lea-Rachel, 2015. "What have economists been doing for the last 50 years? A text analysis of published academic research from 1960-2010," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy (IfW Kiel), vol. 9, pages 1-38.
    14. Matthias Krapf, 2010. "Research evaluation and journal quality weights: Much ado about nothing?," Working Paper Series of the Department of Economics, University of Konstanz 2010-02, Department of Economics, University of Konstanz.
    15. Konstantinos Metaxoglou, 2021. "Canadian Journal of Economics: A historic overview," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 54(3), pages 1418-1453, November.
    16. Pantelis Kalaitzidakis & Theofanis P. Mamuneas & Thanasis Stengos, 2011. "An updated ranking of academic journals in economics," Canadian Journal of Economics, Canadian Economics Association, vol. 44(4), pages 1525-1538, November.
    17. Chia-Lin Chang & Michael McAleer, "undated". "Quality Weighted Citations Versus Total Citations in the Sciences and Social Sciences, with an Application to Finance and Accounting," Documentos de Trabajo del ICAE 2015-01, Universidad Complutense de Madrid, Facultad de Ciencias Económicas y Empresariales, Instituto Complutense de Análisis Económico, revised Jan 2015.
    18. David L. Anderson & John Tressler, 2013. "The Relevance of the “h-” and “g-” Index to Economics in the Context of A Nation-Wide Research Evaluation Scheme: The New Zealand Case," Economic Papers, The Economic Society of Australia, vol. 32(1), pages 81-94, March.
    19. Chia-Lin Chang & Michael Mcaleer, 2014. "Just How Good Are The Top Three Journals In Finance? An Assessment Based On Quantity And Quality Citations," Annals of Financial Economics (AFE), World Scientific Publishing Co. Pte. Ltd., vol. 9(01), pages 1-31.
    20. J. A. García & Rosa Rodriguez-Sánchez & J. Fdez-Valdivia & J. Martinez-Baena, 2012. "On first quartile journals which are not of highest impact," Scientometrics, Springer;Akadémiai Kiadó, vol. 90(3), pages 925-943, March.

    More about this item

    Keywords

    Ranking and Selection; Subset Selection; Multiple Comparisons; Journal Impact Factors Publication Status: Under Review;
    All these keywords.

    JEL classification:

    • A14 - General Economics and Teaching - - General Economics - - - Sociology of Economics
    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:mia:wpaper:2016-08. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Daniela Valdivia (email available below). General contact details of provider: https://edirc.repec.org/data/demiaus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.