[go: up one dir, main page]
IDEAS home Printed from https://ideas.repec.org/p/ipp/wpaper/1010.html
   My bibliography  Save this paper

Knocking down some Assumptions about the Effects of Evaluation Systems on Publications

Author

Listed:
  • Carmen Osuna
  • Laura Cruz Castro
  • Luis Sanz Menéndez
Abstract
In 1989 the Spanish Government established an individual retrospective research evaluation system (RES) for public researchers. This system was voluntary but involved the incentive of a permanent salary bonus. Policy makers have associated the establishment of this evaluation system with the signifi cant increase in the volume of scientifi c publications attributed to Spain over the last decades. In a similar vein to the analyses of other country cases, some scholars have also claimed that the growth of Spain’s international scientifi c publications is a result of the establishment of the new evaluation system. In this paper, we provide a methodological revision of the validity threats in previous research, including some interrupted time series analyses and control groups to investigate the effects of this policy instrument on the number of papers produced by Spanish authors. In the years following the establishment of the evaluation system, the results indicate a considerable increase in the number of papers attributed to Spanish authors among those eligible for evaluation (the 'treated' group), but also in the control groups. After testing various alternative explanations, we conclude that the growth in Spanish publications cannot be attributed indisputably to the effect of the establishment of the RES, but rather to the increase of expenditure and number of researchers in the Spanish R&D system along with some maturation effects. We take this case as an example of the need to improve and refi ne methodologies and to be more cautious when attributing effects to research evaluation mechanisms at the national level.

Suggested Citation

  • Carmen Osuna & Laura Cruz Castro & Luis Sanz Menéndez, 2010. "Knocking down some Assumptions about the Effects of Evaluation Systems on Publications," Working Papers 1010, Instituto de Políticas y Bienes Públicos (IPP), CSIC.
    • Handle: RePEc:ipp:wpaper:1010
    as

    Download full text from publisher

    File URL: http://investigacion.cchs.csic.es/RePEc/ipp/wpaper/10_osuna_cruz_sanz.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Xavier Ballart & Joan Subirats, 1997. "Science and technology policy for a medium-sized industrial country: the case of Spain," Science and Public Policy, Oxford University Press, vol. 24(3), pages 197-205, June.
    2. Robert D. Shelton, 2008. "Relations between national research investment and publication output: Application to an American Paradox," Scientometrics, Springer;Akadémiai Kiadó, vol. 74(2), pages 191-205, February.
    3. Katharine Barker, 2007. "The UK Research Assessment Exercise: the evolution of a national research evaluation system," Research Evaluation, Oxford University Press, vol. 16(1), pages 3-12, March.
    4. Peter Vinkler, 2008. "Correlation between the structure of scientific research, scientometric indicators and GDP in EU and non-EU countries," Scientometrics, Springer;Akadémiai Kiadó, vol. 74(2), pages 237-254, February.
    5. James Adams & Zvi Griliches, 1996. "Measuring Science: An Exploration," NBER Working Papers 5478, National Bureau of Economic Research, Inc.
    6. F. Moya-Anegón & V. Herrero-Solana, 1999. "Science in america latina: A comparison of bibliometric and scientific-technical indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 46(2), pages 299-320, October.
    7. H F Moed & Th N van Leeuwen & M S Visser, 1999. "Trends in publication output and impact of universities in the Netherlands," Research Evaluation, Oxford University Press, vol. 8(1), pages 60-67, April.
    8. Linda Butler, 2003. "Modifying publication practices in response to funding formulas," Research Evaluation, Oxford University Press, vol. 12(1), pages 39-46, April.
    9. Whitley, Richard, 2003. "Competition and pluralism in the public sciences: the impact of institutional frameworks on the organisation of academic science," Research Policy, Elsevier, vol. 32(6), pages 1015-1029, June.
    10. Crespi, Gustavo A. & Geuna, Aldo, 2008. "An empirical study of scientific production: A cross country analysis, 1981-2002," Research Policy, Elsevier, vol. 37(4), pages 565-579, May.
    11. Linda Butler, 2002. "A list of published papers is no measure of value," Nature, Nature, vol. 419(6910), pages 877-877, October.
    12. Arie Rip & Barend J R van der Meulen, 1996. "The post-modern research system," Science and Public Policy, Oxford University Press, vol. 23(6), pages 343-352, December.
    13. Grit Laudel, 2006. "The art of getting funded: How scientists adapt to their funding conditions," Science and Public Policy, Oxford University Press, vol. 33(7), pages 489-504, August.
    14. Mohr, L. B., 2000. "Regression artifacts and other customs of dubious desert," Evaluation and Program Planning, Elsevier, vol. 23(4), pages 397-409, November.
    15. Gomez, I. & Fernandez, M. T. & Zulueta, M. A. & Cami, J., 1995. "Analysis of biomedical research in Spain," Research Policy, Elsevier, vol. 24(3), pages 459-471, May.
    16. Anthony F J van Raan, 1997. "Science as an international enterprise," Science and Public Policy, Oxford University Press, vol. 24(5), pages 290-300, October.
    17. David A. King, 2004. "The scientific impact of nations," Nature, Nature, vol. 430(6997), pages 311-316, July.
    18. Leydesdorff, Loet & Wagner, Caroline S., 2008. "International collaboration in science and the formation of a core group," Journal of Informetrics, Elsevier, vol. 2(4), pages 317-325.
    19. Butler, Linda, 2003. "Explaining Australia's increased share of ISI publications--the effects of a funding formula based on publication counts," Research Policy, Elsevier, vol. 32(1), pages 143-155, January.
    20. Georghiou, Luke, 1998. "Global cooperation in research," Research Policy, Elsevier, vol. 27(6), pages 611-626, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Carmen Osuna & Laura Cruz-Castro & Luis Sanz-Menéndez, 2011. "Overturning some assumptions about the effects of evaluation systems on publication performance," Scientometrics, Springer;Akadémiai Kiadó, vol. 86(3), pages 575-592, March.
    2. Leydesdorff, Loet & Wagner, Caroline, 2009. "Macro-level indicators of the relations between research funding and research output," Journal of Informetrics, Elsevier, vol. 3(4), pages 353-362.
    3. Auranen, Otto & Nieminen, Mika, 2010. "University research funding and publication performance--An international comparison," Research Policy, Elsevier, vol. 39(6), pages 822-834, July.
    4. R. Inglesi-Lotz & A. Pouris, 2013. "The influence of scientific research output of academics on economic growth in South Africa: an autoregressive distributed lag (ARDL) application," Scientometrics, Springer;Akadémiai Kiadó, vol. 95(1), pages 129-139, April.
    5. Pin-Hua Lin & Jong-Rong Chen & Chih-Hai Yang, 2014. "Academic research resources and academic quality: a cross-country analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 109-123, October.
    6. Tanel Hirv, 2022. "The interplay of the size of the research system, ways of collaboration, level, and method of funding in determining bibliometric outputs," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(3), pages 1295-1316, March.
    7. Mehdi Rhaiem & Nabil Amara, 2020. "Determinants of research efficiency in Canadian business schools: evidence from scholar-level data," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 53-99, October.
    8. Roula Inglesi-Lotz & Mehmet Balcilar & Rangan Gupta, 2014. "Time-varying causality between research output and economic growth in US," Scientometrics, Springer;Akadémiai Kiadó, vol. 100(1), pages 203-216, July.
    9. van den Besselaar, Peter & Heyman, Ulf & Sandström, Ulf, 2017. "Perverse effects of output-based research funding? Butler’s Australian case revisited," Journal of Informetrics, Elsevier, vol. 11(3), pages 905-918.
    10. Frenken, Koen & Hardeman, Sjoerd & Hoekman, Jarno, 2009. "Spatial scientometrics: Towards a cumulative research program," Journal of Informetrics, Elsevier, vol. 3(3), pages 222-232.
    11. Sandström, Ulf & Van den Besselaar, Peter, 2018. "Funding, evaluation, and the performance of national research systems," Journal of Informetrics, Elsevier, vol. 12(1), pages 365-384.
    12. Alonso Rodríguez-Navarro & Francis Narin, 2018. "European Paradox or Delusion—Are European Science and Economy Outdated?," Science and Public Policy, Oxford University Press, vol. 45(1), pages 14-23.
    13. Inglesi-Lotz, R., 2019. "Energy research and R&D indicators: An LMDI decomposition analysis for the IEA Big 5 in energy research," Energy Policy, Elsevier, vol. 133(C).
    14. Elham Erfanian & Amir B. Ferreira Neto, 2017. "Scientific output: labor or capital intensive? An analysis for selected countries," Scientometrics, Springer;Akadémiai Kiadó, vol. 112(1), pages 461-482, July.
    15. Ricardo Arencibia-Jorge & Felix Moya-Anegón, 2010. "Challenges in the study of Cuban scientific output," Scientometrics, Springer;Akadémiai Kiadó, vol. 83(3), pages 723-737, June.
    16. Aparna Basu, 2014. "The Albuquerque model and efficiency indicators in national scientific productivity with respect to manpower and funding in science," Scientometrics, Springer;Akadémiai Kiadó, vol. 100(2), pages 531-539, August.
    17. Chadi Azmeh, 2022. "Quantity and quality of research output and economic growth: empirical investigation for all research areas in the MENA countries," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(11), pages 6147-6163, November.
    18. Yong Yi & Wei Qi & Dandan Wu, 2013. "Are CIVETS the next BRICs? A comparative analysis from scientometrics perspective," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(2), pages 615-628, February.
    19. Wen-Chi Hung & Ling-Chu Lee & Min-Hua Tsai, 2009. "An international comparison of relative contributions to academic productivity," Scientometrics, Springer;Akadémiai Kiadó, vol. 81(3), pages 703-718, December.
    20. Ryan Zelnio, 2012. "Identifying the global core-periphery structure of science," Scientometrics, Springer;Akadémiai Kiadó, vol. 91(2), pages 601-615, May.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ipp:wpaper:1010. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Adelheid Holl (email available below). General contact details of provider: https://edirc.repec.org/data/ippcses.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.