Stochastic Trend

I have a new paper, coauthored with Ida Kubiszewski, Bob Costanza, and Luke Concollato, which investigates the development of the field of ecosystem services over the last decade since the founding of the journal Ecosystem Services. This is an open access publication – my first in a so-called hybrid journal. We used the University College London read and publish agreement with Elsevier to publish the paper. ANU now has a similar agreement starting in 2023.

The paper updates Ida and Bob's paper published in 2012: "The authorship structure of ‘‘ecosystem services’’ as a transdisciplinary field of scholarship". In this paper, we update and expand that analysis and compare results with those we found in the previous analysis. We also analyse the influence that the journal Ecosystem Services has had on the field over its first 10 years. We look at which articles have had the most influence on the field (as measured by the number of citations in Ecosystem Services) and on the broader scientific literature (as measured by total citations). We also look at how authorship networks, topics, and the types of journals publishing on the topic have changed. 

Not surprisingly, there has been significant growth in the number of authors (12,795 to 91,051) and number of articles published (4,948 to 33,973) on ecosystem services since 2012. Authorship networks have also expanded significantly, and the patterns of co-authorship have evolved in interesting ways. The most prolific authors are no longer in as tight clusters as they were 10 years ago.

The network chart shows the coauthorship relations among the 163 most prolific authors – those authors who have published more than 30 articles in the field. Colors indicate continent: Yellow = North America, red = South America, blue = Europe, purple – Africa, green = Asia, and orange = Oceania. The greatest number of authors is in Europe and they almost all collaborate with other top authors. Only in Asia and to a lesser degree North America are there top authors who do not collaborate with other top authors.

Costanza et al. (1997) is the most influential article in terms of citations in the journal Ecosystem Services and "Global estimates of the value of ecosystems andtheir services in monetary units" by de Groot et al. (2012) is now the most cited article published in Ecosystem Services.

Ecosystem Services is now the most prolific publisher of articles on ecosystem services among all the journals that have published in the area. There are nine journals that are both on the list of the 20 journals cited most often in Ecosystem Services and on the list of the top 20 journals cited by articles published in Ecosystem Services: Ecosystem Services, Ecological Economics, Ecological Indicators, Science of the Total Environment, Land Use Policy, Journal of Environmental Management, PLoS One, Ecology and Society, and Environmental Science & Policy.

For the first time in a decade, I updated my spreadsheet on downloads and abstract views per person and per item on RePEc.

The downward trends I identified ten years ago have continued, though there was an uptick during the pandemic, which has now dissipated. There was more of an increase in abstract views than in downloads in the pandemic.

Since the end of 2011 both abstract views and downloads per paper have fallen by about 80%. Total papers rose by around 260%, while total downloads fell 38% and total abstract views 27%. 

I'd guess that a mixture of the explanatory factors I suggested last time has continued to be in play.

I have a new working paper coauthored with Johannes König and Richard Tol. It's a follow up to my 2013 paper in the Journal of Economic Literature, where I computed standard errors for simple journal impact factors for all economics journals and tried to evaluate whether the differences between journals were significant.* In the new paper, we develop standard errors and confidence intervals for recursive journal impact factors, which take into account that some citations are more prestigious than others, as well as for the associated ranks of journals. We again apply these methods to the all economics journals included in the Web of Science.

Recursive impact factors include the popular Scimago Journal Rank, or SJR, and Clarivate's Article Influence score. We use Pinski and Narin's invariant method, which has been used in some rankings of economics journals. 

As simple impact factors are just the mean citations an article published in a journal in a given period receives in a later year, it is easy to compute standard errors for them using the formula for the standard error of the mean. But the vector of recursive impact factors is the positive eigenvector of a matrix and its variance does not have a simple analytical form.

So, we use bootstrapping to estimate the distribution of each impact factor. Taking all 88,928 articles published in 2014-18 in the economics journals included in the Web of Science, we resample from this dataset and compute the vector of recursive impact factors from the new dataset.** Repeating this 1,000 times we pick the 2.5% or 97.5% range of values for each journal to get a 95% confidence interval:

95% confidence intervals of the recursive impact factor, arithmetic scale (left axis) and logarithmic scale (right axis).

The graph repeats the same data twice with different scales so that it's possible to see some detail for both high- and low-ranked journals. Also, notice that while the confidence intervals for the highest ranked journals are quite symmetric, confidence intervals become increasingly asymmetric as we go down the ranks. 

The top ranked journal, the Quarterly Journal of Economics, clearly stands out above all others. The confidence intervals of all other journals overlap with those of other journals and so the ranks of these journals are somewhat uncertain.*** So, next we construct confidence intervals for the journals' ranks.

It turns out that there are a few ways to do this. We could just construct a journal ranking for each iteration of the bootstrap and then derive the distribution of ranks for each individual journal across the 1,000 iterations. Hall and Miller (2009), Xie et al. (2009), and Mogstad et al. (2020) show that this procedure may not be consistent when some of the groups (here journals) being ranked are tied or close to tied. The corrected confidence intervals are generally broader than the naive bootstrap approach.

We compute confidence intervals for ranks using the simple bootstrap, the Xie et al. method, and the Mogstad et al. method:

 

95% confidence intervals of the rank based on the recursive impact factor. The inner intervals are based on Goldstein’s bootstrap method, the middle intervals use Xie’s correction to the bootstrap, and the outer intervals follow Mogstad’s pairwise comparison.

The simple bootstrap under-estimates the true range of ranks, while it seems that the Mogstad et al. method might be overly conservative. On the other hand, Xie et al.' s approach depends on choosing a couple of "tuning parameters".  

All methods agree that the confidence interval of the rank of the Quarterly Journal of Economics only includes one. Based on the simple bootstrap, the remainder of the "Top-5'' journals are in the top 6 together with the Journal of Finance, while the Xie et al. method and the Mogstad et al. methods generally broaden estimated confidence intervals, particularly for mid-ranking journals. All methods agree that most apparent differences in journal quality are, in fact, mostly insignificant. We think that impact factors, whether simple or recursive should always be published together with confidence intervals.

* The latter exercises were a bit naive. As pointed out by Horrace and Parmeter (2017), we need to account for the issue of multiple comparisons. 

** Previous research on this topic resampled at the journal level, missing most of the variation in citation counts.

*** Overlapping confidence intervals don't neccessarily mean that there is no signficant difference between two means. Goldstein and Harvey (1995) show that the correct confidence intervals for such a test of the difference between two means are narrower than the conventional 95% confidence intervals. On the other hand, for multiple comparisons we would want wider confidence intervals.

I have a new paper with my Crawford School colleague Bjoern Dressel published in Asia & the Pacific Policy Studies (open access). The data and figures for the article are on Figshare. Bjoern has been interested for a while in ranking public policy schools in the Asia-Pacific region.  But a comprehensive ranking seemed hard to achieve. Recently, I came across an article by Ash and Urquiola (2020) that ranks US public policy schools according to their research output and impact. Well, we thought, if they can rank schools just by their research output and not by their education and public policy impact then so can we 😀. Research is the easiest component to evaluate.

We compare the publication output of 45 schools with at least one publication listed in Scopus between 2014 and 2018, based on affiliations listed on the publications rather than current faculty. We compute the 5-Year impact factor for each school. This is identical to the impact factor reported for academic journals, but we compute it for a school rather than a journal. It is the mean number of citations received in 2019 by a publication published between 2014 and 2018. This can be seen as an estimate of research quality. We also report the standard error of the impact factor as in my 2013 article in the Journal of Economic Literature. If we treat the impact factor as an estimate of the research quality of a school then we can construct a confidence interval to express how certain or uncertain we are about that estimate. This graph shows the schools ranked by impact factor with a 90% confidence interval:

Peking and Melbourne are the two top-ranked schools but the point estimates have a very wide confidence interval. This is because their research output is relatively small and the variance of citations is quite large. The third ranked school – SGPP in Indonesia – only had two publications in our target period. After that there are several schools with much narrower confidence intervals. These mostly have more publications.

Here we can see the impact factors on the y-axis and the number of publications of each school on the x-axis. Three schools clearly stand out at the right: Crawford, Lee Kwan Yew, and Tsinghua. These schools are also top-ranked by total citations, which combines the quality and quantity variables. The three top schools account for 54% of publications and 63% of citations from the region.

In general, the elite schools are in China and Australia. Australia has three out of the top ten schools ranked by impact factor and total citations, despite its small population size. China, on the other hand has at least five schools ranked in the top ten across both rankings, which is remarkable given that many of these schools have been established only in the last 15 years (though linked to well-established research universities).

We found more schools that had no publications in Scopus in the target period. Perhaps in some cases they are too new, or faculty use their other affiliations, but clearly there is a lot of variation in research-intensiveness. Somewhat surprising is the low ranking of public policy schools in Japan and India – both countries with a considerable number of public policy schools, but none in the top ten schools when ranked by 5-year citation impact factor or total number of citations. 

One reason for the strong performance of the Chinese schools is that they focus to some degree on environmental issues, and particularly climate change, where citation numbers tend to be higher. We did not adjust for differences in citations across fields in this research, but this is something that future research should address.

That is the title of a paper in the American Economic Review by Motty Perry and Philip Reny. They present five axioms that they argue a good index of individual citation performance should conform to. They show that the only index that satisfies all five axioms is the Euclidean length of the list of citations to each of a researcher's publications – in other words, the square root of the sum of squares of the citations to each of their papers.* This index puts much more weight on highly cited papers and much less on little cited papers than simply adding up a researcher's total citations would. This is a result of their "depth relevance" axiom. A citation index that is depth relevant always increases when some of the citations of a researcher's less cited papers are instead transferred to some of the researcher's more cited papers. In the extreme, it rewards "one hit wonders" who have a single highly cited paper, over consistent performers who have a more extensive body of work with the same total number of citations.

The Euclidean index is an example of what economists call constant elasticity of substitution, or CES, functions. Instead of squaring each citation number, we could raise it to a different power, such as 1.5, 0.5, or anything else. Perry and Reny show that the rank correlation between the National Research Council peer-reviewed ranks of the top 50 U.S. economics departments and the CES citation indices of the faculty employed in those departments is at a maximum for a power of 1.85:


This is close to 2 and suggests that the market for economists values citations in a similar way to the Euclidean index.

RePEc acted unusually quickly to add this index to their rankings. Richard Tol and I have a new working paper that discusses this new citation metric. We introduce an alternative axiom: "breadth relevance", which rewards consistent achievers. This axiom states that a citation index always increases when some citations from highly cited papers are shifted to less cited papers. We also reanalyze the dataset of economists at the top 50 U.S. departments that Perry and Reny looked at and a much larger dataset that we scraped from CitEc for economists at the 400 international universities ranked by QS. Unlike Perry and Reny, we take into account the fact that citations accumulate over a researcher's career and so junior researchers with few citations aren't necessarily weaker researchers than senior researchers with more citations. Instead, we need to compare citation performance within each cohort of researchers measured by the years since they got their PhD or published their first paper.

We show that a breadth relevant index that also satisfies Perry and Reny's other axioms is a CES function with exponent of less than one. Our empirical analysis finds that the distribution of economists across departments is in fact explained best by the simple sum of their citations, which is equivalent to a CES function with exponent of one, that favors neither depth nor breadth. However, at lower ranked departments – departments ranked by QS from 51 to 400 – the Euclidean index does explain the distribution of economists better than does total citations.


In this graph, the full sample is the same dataset that Perry and Reny used in their graph. The peak correlation is for a lower exponent – tau or sigma** – simply because we take into account cohort effects by computing the correlation for a researcher's citation index relative to the cohort mean.*** While the distribution across the top 25 departments is similarly to the full sample, with a peak at a slightly lower exponent that is very close to one, we don't find any correlation between citations and department rank for the next 25 departments. It seems that there aren't big differences between them.

Here are the correlations for the larger dataset that uses CitEc citations for the 400 universities ranked by QS:


For the top 50 universities, the peak correlation is for an exponent of 1.39 but for the next 350 universities the peak correlation is for 2.22. The paper also includes parametric maximum likelihood estimates that come to similar conclusions.

Breadth per se does not explain the distribution of researchers in our sample, but the highest ranked universities appear to weight breadth and depth equally, while lower-ranked universities do focus on depth, giving more weight to a few highly cited papers.

A possible speculative explanation of behavior across the spectrum of universities could be as follows. Lowest-ranked universities, outside of the 400 universities ranked by QS, might simply care about publication without worrying about impact. Having more publications would be better than having fewer at these institutions, suggesting a breadth relevant citation index. Our exploratory analysis that includes universities outside of those ranked by QS supports this. We found that breadth was inversely correlated with average citations in the lower percentiles.

Middle-ranked universities, such as those ranked between 400 and 50 in the QS ranking, care about impact; having some high-impact publications is better than having none and a depth-relevant index describes behavior in this interval. Finally, among the top-ranked universities such as the QS top 50 or NRC top 25, hiring and tenure committees wish to see high-impact research across all of a researcher's publications and the best-fit index moves towards. Here, adding lower-impact publications to a publication list that contains high-impact ones is seen as a negative.

* As monotonic transformations of the index also satisfy the same axioms, the simplest index that satisfies the axioms is simply the sum of squares.

** In the paper, we refer to an exponent of less than one as tau and an exponent greater than one as sigma.

*** The Ellison dataset that Perry and Reny use, uses Google Scholar data and truncates each researcher's publication list at 100 papers. With all working paper variants, it's not hard to exceed 100 items. This could bias the analysis in favor of depth rather than breadth. We think that the correlation computed for researchers with 100 papers or less only is a better way to test whether depth or breadth best explains the distribution of economists across departments. The correlation peaks very close to one for this dataset.

As part of our self-evaluation for the upcoming review of the Crawford School, I have been doing some bibliometric analysis. One thing I have come up with is calculating an impact factor for the School and some comparator institutions. This is easy to do in Scopus. It's the same idea as computing one for an individual or a journal, of course. I am using a 2016, 5 year impact factor. Just get total citations in 2016 to all articles and reviews published in 2011-2015. Divide by the number of articles. Here are the results with 95% confidence intervals:


The main difficulty I had was retrieving articles for some institutions such as the School of Public Affairs at Sciences Po. Very few articles came back for various variants of the name that I tried. I suspect that faculty are using departmental affiliations. I had a similar problem with IPA at LSE. So, I report the whole of LSE in the graph. It is easy to understand this metric in comparison to journal impact factors. As an individual metric the confidence interval will usually be large, though my 2016 impact factor was 5.9 with a 4.2 to 7.5 confidence interval. That's more precise than the estimate for SIPA.

Since it was introduced in late 2004, Google Scholar has rapidly grown to become a widely used tool for finding and assessing the impact of academic literature. The database still suffers from noise relative to its competitors Scopus and Web of Science but it has broader coverage, especially in the social sciences and humanities and is open access. As the database developed, Google have periodically added new information sources to the database. This resulted in a rapid growth in estimated citations of articles in the early years. However, it now seems that the database has matured. The following graph shows the growth rate of citations to my research in the previous 12 months, measured monthly since 2009 for Google Scholar in blue and a bit more intermittently for Scopus in red. I have also fitted exponential trend lines to the two series:

Initially the growth rate of Google Scholar citations was very high and very erratic. But the month to month variation in the annual growth rate has reduced drastically over time. By contrast, the growth rate of Scopus citations has been much more consistent, with a slow rate of decline in the percentage growth rate over time. Interestingly, the two series have also converged to a common growth rate of 17-18% per year. So, it seems that Google's database is now as mature as Scopus is. This doesn't mean that Google is now as high a quality data source as Scopus is. It isn't. But large revisions to citation counts or additions of large new data sources seems to be a thing of the past.