Results of the Sherpa FACT accuracy testing – 95% accurate

UKCoRR welcomes the results of a  recent exercise – undertaken by UK librarians, repository managers and Sherpa Services – that has shown that the results produced by SHERPA/FACT (Funders & Authors Compliance Tool) have an accuracy rate of over 95%

The FACT service was developed to help researchers get a simple answer to the question “does this journal have an open access publishing policy compliant with my funder’s open access mandate?”. The FACT service – which draws its information from the SHERPA/ROMEO and SHERPA/JULIET databases – seeks to provide a yes/no answer to this question, as well as providing information about how an author can comply with a funder policy.

There had been some discussion at the SHERPA/FACT Advisory Board, raised by UKCoRR members and their institutions – as to whether the information provided by FACT was accurate.

To address this issue an exercise was undertaken – by members of UKCoRR – to manually check a statistically significant number of journal/funder combinations and then compare the information this group had found with the information provided by FACT. Where the independent reviewers arrived at a different conclusion to that provided by FACT, then that journal/publisher combination was subjected to detailed and exhaustive investigation to arrive at an evidenced answer.

At the end of this exercise, it was found that the FACT service provides correct information in over 95% of cases.

The study clearly highlighted the difficulties that even highly experienced repository staff have at deciphering publisher OA policies. Indeed, the initial testing undertaken by UKCoRR members suggested that FACT was only accurate on 57% of occasions. When these journal/funder combinations were investigated further, however, close examination of the often complex conditions and the interactions between different statements and policies showed that FACT was correct in almost all of the cases.

The SHERPA/FACT team as well as the SHERPA/FACT Advisory Group would like to extend their thanks to the UKCoRR Members who took part in this checking process for their time commitment as well as their extensive knowledge of this area of work. This exercise has proved that the SHERPA/FACT service can be relied upon as a source of advice for UK researchers. UKCoRR also encourages it’s members to continue to communicate with SHERPA/FACT where discrepancies are found to continue to improve the quality of the information SHERPA/FACT relies upon.

Issues of interpretation and the interaction between the various policies have been seen to be the key to the discrepancy between our manually checked results and Sherpa’s findings. There is further work to be done here and UKCoRR looks forward to continuing to work with the SHERPA/FACT Advisory Group to develop increased clarity in this area.

To see the full data and study methodology, visit Figshare.

The study was commissioned by the SHERPA/FACT Advisory Board – which includes representatives from UKCoRR, Jisc, Wellcome Trust, Research Councils UK (RCUK), CRC Nottingham, Association of Learned & Professional Society Publishers (ALPSP), Higher Education Funding Council for England (HEFCE), Publishers Association and SCONUL.

A blog post from Jisc on this project is available as well as a press release on the project.

Ranking altmetrics: DIY

In the previous post Network effects: on alternative metrics I presented the top 10 articles by altmetric for repository downloads in April 2015 as recorded by IRUS-UK. This is a quick post on how to do it for any set of DOIs.

IRUS-UK provides a rich data set currently comprising 84 UK repositories with a total of more than 36 million unique item downloads recorded, and if you are an IRUS participant you can easily derive data for your own repository.

Log-in to the portal at via UK Access Management Federation and navigate to Statistics Reports, Article Report 4, limit the date range you are interested in, select your repository – I’m actually using White Rose Research Online here as a consortium of 3 major research institutions (Leeds, Sheffield, York)- and output as CSV:

Screenshot from IRUS-UK
Screenshot from IRUS-UK

Use MS Excel to filter blanks and other gubbins from the DOI column and cut and paste the resulting list into a Google sheet.

Now you need the script below, kindly provided by the founder of Euan Adie, Tools -> Script editor -> Blank project, just cut and paste and save the project:

function ALTMETRIC_SCORE(input) {
var response = UrlFetchApp.fetch(“” + input, {muteHttpExceptions: true});
var data = response.getContentText(“UTF-8″);
if (data.length <= 10) {return 0;}
var json = JSON.parse(data);
return Math.floor(json.score);


Now return to your Google Sheet and simply type the formula into the first cell of column B referencing the equivalent cell in column A:

1. Type the formula into the first cell as =ALTMETRIC_SOURCE(A2)
1. Type the formula into the first cell as =ALTMETRIC_SCORE(A2)

Now you just need to drag the formula into every cell in column B – using the handle in the bottom right corner – to run against the corresponding cell with a DOI in column A.

HOWEVER, unfortunately you can’t just do thousands or even hundreds at a time as it will upset Google’s rate limiter and you will get increasing numbers of cells screaming ERROR! so you need to do them in batches of about 100 at a time:

Drag formula into all cells in column B
2. Drag formula into all cells in column B
3. Script will run for each cell in column A
3. Script will run for each cell in column A
4. Don't drag to the next batch of cells until data is loaded
4. Don’t drag to the next batch of cells until data is loaded

Once this is done, which can be painstaking with several thousand records, you can use Sort sheet Z-A to rank highest to lowest.

WARNING – if you do this with the column containing the live formula it will run again for all cells at once and throw hundreds of errors which is what the painstaking process was meant to avoid…so copy and paste values only into the next column and sort on that! It is likely that there will be some ERRORS and you can run the formula again in another column though some of them may just be dud DOIs.

And there they are, all records downloaded from WRRO (with DOIs*) since that repository joined IRUS in October 2013 ranked highest to lowest by altmetric *some of the records excluded by the initial filter on DOI may in fact have a DOI of course that isn’t available from IRUS (or presumably the originating repository) so to be really thorough you might try to track them down!

Finally, altmetric donuts can be embedded easily using code provided at so here are the top ten from WRRO:

  1. Lin, J, Pan, D, Davis, SJ, Zhang, Q, He, K, Wang, C, Streets, DG, Wuebbles, DJ and Guan, D (2014) China’s international trade and air pollution in the United States. Proceedings of the National Academy of Sciences of the United States of America, 111 (5). 1736 – 1741. ISSN 0027-8424

    DOI lookup(Available OA from publisher website: licence unclear)
    Total downloads from WRRO: 1*
    (* Full text not actually in WRRO, didn’t think IRUS counted links and also seems unlikely that only ever linked once.)

  2. Brienen, RJW, Phillips, OL, Feldpausch, TR, Gloor, E, Baker, TR, Lloyd, J, Lopez-Gonzalez, G, Monteagudo-Mendoza, A, Malhi, Y, Lewis, SL, Vásquez Martinez, R, Alexiades, M, Álvarez Dávila, E, Alvarez-Loayza, P, Andrade, A, Aragão, LEOC, Araujo-Murakami, A, Arets, EJMM, Arroyo, L, Aymard C, GA, Bánki, OS, Baraloto, C, Barroso, J, Bonal, D, Boot, RGA, Camargo, JLC, Castilho, CV, Chama, V, Chao, KJ, Chave, J, Comiskey, JA, Cornejo Valverde, F, da Costa, L, de Oliveira, EA, Di Fiore, A, Erwin, TL, Fauset, S, Forsthofer, M, Galbraith, DR, Grahame, ES, Groot, N, Hérault, B, Higuchi, N, Honorio C, E, Keeling, H, Killeen, TJ, Laurance, WF, Laurance, S, Licona, J, Magnussen, WE, Marimon, BS, Marimon-Junior, BH, Mendoza, C, Neill, DA, Nogueira, EM, Núñez, P, Pallqui Camacho, NC, Parada, A, Pardo-Molina, G, Peacock, J, Peña-Claros, M, Pickavance, GC, Pitman, NCA, Poorter, L, Prieto, A, Quesada, CA, Ramírez, F, Ramírez-Angulo, F, Restrepo, Z, Roopsind, A, Rudas, A, Salomão, RP, Schwarz, M, Silva, N, Silva-Espejo, JE, Silveira, M, Stropp, J, Talbot, J, ter Steege, H, Teran-Aguilar, J, Terborgh, J, Thomas-Caesar, R, Toledo, M, Torello-Raventos, M, Umetsu, RK, van der Heijden, GMF, van der Hout, P, Guimarães Vieira, IC, Vieira, SA, Vilanova, E, Vos, VA and Zagt, RJ (2015) Long-term decline of the Amazon carbon sink. Nature, 519 (7543). 344 – 348. ISSN 0028-0836

    DOI lookup(Access restricted from publisher website)
    Total downloads from WRRO: 2*
    (* embargoed from WRRO – unclear how can have any downloads.)

  3. Bond, TC, Doherty, SJ, Fahey, DW, Forster, PM, Berntsen, T, DeAngelo, BJ, Flanner, MG, Ghan, S, Kärcher, B, Koch, D, Kinnell, S, Kondo, Y, Quinn, PK, Sarofim, MC, Schultz, MG, Schulz, M, Venkataraman, C, Zhang, H, Zhang, S, Bellouin, N, Guttikunda, SK, Hopke, PK, Jacobson, MZ, Kaiser, JW, Klimont, Z, Lohmann, U, Schwarz, JP, Shindell, D, Storelvmo, T, Warren, G and Zender, CS (2013) Bounding the role of black carbon in the climate system: A scientific assessment. Journal of Geophysical Research: Atmospheres, 118 (11). 5380 – 5552.

    DOI lookup(Available OA from publisher website: CC-BY-NC-ND)
    Total downloads from WRRO: 102

  4. McMillan, M, Shepherd, A, Sundal, A, Briggs, K, Muir, A, Ridout, A, Hogg, A and Wingham, D (2014) Increased ice losses from Antarctica detected by CryoSat-2. Geophysical Research Letters, 41 (11). 3899 – 3905. ISSN 0094-8276

    DOI lookup(Access restricted from publisher website: CC-BY*)
    Total downloads from WRRO: 12
    (* restricted access presumably in error)

  5. Marotzke, J and Forster, PM (2015) Forcing, feedback and internal variability in global temperature trends. Nature, 517 (7536). 565 – 570. ISSN 0028-0836

    DOI lookup(Access restricted from publisher website)
    Total downloads from WRRO: 1

  6. Otto, A, Otto, FEL, Allen, MR, Boucher, O, Church, J, Hegerl, G, Forster, PM, Gillett, NP, Gregory, J, Johnson, GC, Knutti, R, Lohmann, U, Lewis, N, Marotzke, J, Stevens, B, Myhre, G and Shindell, D (2013) Energy budget constraints on climate response. Nature Geoscience, 6 (6). 415 – 416. ISSN 1752-0894

    DOI lookup(Access restricted from publisher website)
    Total downloads from WRRO: 537

  7. Challinor, AJ, Watson, J, Lobell, DB, Howden, SM, Smith, DR and Chhetri, N (2014) A meta-analysis of crop yield under climate change and adaptation. Nature Climate Change, 4. 287 – 291. ISSN 1758-678X

    DOI lookup(Access restricted from publisher websiter)
    Total downloads from WRRO: 77

  8. MacPherson, Hugh, Richmond, Stewart John, Bland, Martin, Brealey, Stephen Derek, Gabe, Rhian, Hopton, Ann Kathryn, Keding, Ada, Lansdown, Harriet, Perren, Sara, Sculpher, Mark, Spackman, Eldon, Torgerson, David John and Watt, Ian (2013) Acupuncture and counselling for depression in primary care : a randomised controlled trial. PLoS Medicine. 1001518. ISSN 1549-1277

    DOI lookup(Available OA from publisher website: CC-BY)
    Total downloads from WRRO: 52

  9. Schroeder, J., Dugdale, H.L., Radersma, R., Hinsch, M., Buehler, D.M., Saul, J., Porter, L., Liker, A., De Cauwer, I., Johnson, P.J., Santure, A.W., Griffin, A.S., Bolund, E., Ross, L., Webb, T.J., Feulner, P.G.D., Winney, I., Szulkin, M., Komdeur, J., Versteegh, M.A., Hemelrijk, C.K., Svensson, E.I., Edwards, H., Karlsson, M., West, S.A., Barrett, E.L.B., Richardson, D.S., van den Brink, V., Wimpenny, J.H., Ellwood, S.A., Rees, M., Matson, K.D., Charmantier, A., dos Remedios, N., Schneider, N.A., Teplitsky, C., Laurance, W.F., Butlin, R.K. and Horrocks, N.P.C. (2013) Fewer invited talks by women in evolutionary biology symposia. Journal Of Evolutionary Biology, 26 (9). pp. 2063-2069. ISSN 1420-9101

    DOI lookup(Available OA from publisher website: CC-BY)
    Total downloads from WRRO: 6

  10. Jetz, W, Thomas, G.H, Joy, J.B, Redding, D.W, Hartmann, K and Mooers, A.O (2014) Global Distribution and Conservation of Evolutionary Distinctness in Birds. Current Biology, 24 (9). pp. 919-930.

    DOI lookup(Available OA from publisher website: CC-BY)
    Total downloads from WRRO: 91

Network effects: on alternative metrics

“No one can read everything. We rely on filters to make sense of the scholarly literature, but the narrow, traditional filters are being swamped. However, the growth of new, online scholarly tools allows us to make new filters; these altmetrics reflect the broad, rapid impact of scholarship in this burgeoning ecosystem. We call for more tools and research based on altmetrics.”

altmetrics: a manifesto

Research metrics are nothing if not controversial, none more so than the “alternative” variety currently in vogue from Plum Analytics, ImpactStory and the donut people themselves For the most part, the twitterati are buying into them in a big way, not to mention Big Publishers and Open Access advocates, though some influential voices are dismissive. At best. See this title from David Colquhoun and Andrew Plested “Why you should ignore altmetrics and other bibliometric nightmares“. Or even read it – it’s hard to deny that most folk who retweet the monster-altmetric articles probably don’t (personally I eschew the RT without having read an article, whether academic or pop-cultural trivia. Maybe that’s just me*)

* I’ll admit occasionally to having retweeted *before* reading but only if I trust the source.

Though a mere bibliometric tinkerer myself, I rather take umbrage on behalf of professional bibliometricians everywhere who are somewhat maligned in this article: “The mistake made by all bibliometricians is that they fail to consider the content of papers, because they have no desire to understand research.” This arguably needs a citation and at least one cohort of bibliometricians to take an IQ test to establish if bibliometrics really are “for people who aren’t prepared to take the time (or lack the mental capacity) to evaluate research by reading about it.” I can’t necessarily back up with a citeable source myself but I’m willing to bet that the very first thing one learns as a bibliometrician is the strengths and, often profound, weaknesses of the various metrics, whether JIF, h-index or alternative. To dismiss their value entirely seems to rather throw the baby out with the bathwater.

Not that I don’t have reservations – the colour coded donut of surrounding a number arguably lacks clarity and I’m not entirely certain what the number actually refers to, though it seems to correlate vaguely with tweets (coloured blue in the donut) which is invariably the highest “metric” – and arguably the least valuable – there is presumably some weighting towards potentially more valuable, less ephemeral activity on blogs (yellow) and in mainstream news outlets (dark red). I’m not convinced Mendeley and CiteULike necessarily reflect representative networks across all disciplines (dark red and light blue respectively) or Facebook for that matter (dark blue)…however, this is surely the point, the network, and while altmetrics do perhaps pander to a human propensity to quantify with raw numbers – though no more than traditional citations – their main strength is undoubtedly to visualise and leverage the network; to connect with peers online, or with interested laypersons, to foster community and promote research activity and impact beyond the walls of the ivory tower.

N.B. I’ve just picked up this article from Twitter – Numbers behind Numbers: The Altmetric Score and Sources Explained…but haven’t read it yet!

Donuts can be embedded easily using code provided at and the API can be accessed free of charge, with various paid options for more sophisticated functionality, though with a little knowhow you can derive useful data without a subscription to the premium service. There is also a plugin available from the EPrints Bazaar


Below are the top 10 articles by altmetric for repository downloads in April 2015 as recorded by IRUS-UK and derived from 84 UK repositories and a total of more than 87,000 unique items. Data is presented with a link to the primary repository and highlights numbers of total downloads* from this, and any other repositories where the paper is archived (*i.e. all downloads from each repository, not just April). There is a DOI lookup link and a note whether the article is available to download from the publisher’s site (N.B. only 3 of these 10 are restricted access with 5 under CC-By and 2 freely accessible albeit © All Rights Reserved).

No further analysis is offered other than to say that there does not appear to be any obvious correlation between altmetrics and repository downloads – number 1 is obviously related to a major story in the media about a certain King found in a Leicester carpark – but, it bears reiterating, that is not necessarily the point; rather repositories are an embedded, institutionally managed element of researchers’ online network with tools like IRUS and altmetrics enabling data to be visualised and leveraged across an increasingly coherent infrastructure.

N.B. Information on how this data was derived using the export function from IRUS-UK and a Google Sheet will follow in a subsequent post.

  1. King, T.E., Fortes, G.G., Balaresque, P., Thomas, M.G., Balding, D., Delser, P.M., Neumann, R., Parson, W., Knapp, M., Walsh, S., Tonasso, L., Holt, J., Kayser, M., Appleby, J., Forster, P., Ekserdjian, D., Hofreiter, M., Schürer, K. (2014) Identification of the remains of King Richard III. Nature Communications, 5, art. no. 5631

    DOI lookup (Available OA from publisher: CC-BY)
    Total downloads from Leicester Research Archive to April 2015:

  2. Sumner, Petroc, Vivian-Griffiths, Solveiga, Boivin, Jacky, Williams, Andrew James, Venetis, C. A., Davies, Aimee, Ogden, Jack, Whelan, Leanne, Hughes, Bethan, Dalton, Bethan, Boy, Frederic and Chambers, Christopher D. 2014. The association between exaggeration in health related science news and academic press releases: retrospective observational study. BMJ 349, g7015

    DOI lookup (Available OA from publisher: CC-BY)
    Total downloads from Online Research @ Cardiff (ORCA) to April 2015: 42

  3. ATLAS collaboration (2012) Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC. Physics Letters B, Volume 716 (Number 1). pp. 1-29. ISSN 0370-2693

    DOI lookup (Available OA from publisher: CC-BY)
    Total downloads from
    Lancaster EPrints to April 2015: 29
    Total downloads from
    WRAP: Warwick Research Archive Portal to April 2015: 16

  4. RTS,S Clinical Trials Partnership; (2011) First results of phase 3 trial of RTS,S/AS01 malaria vaccine in African children. The New England journal of medicine, 365 (20). pp. 1863-75. ISSN 0028-4793

    DOI lookup (Freely available from publisher: © Massachusetts Medical Society) 
    Total downloads from LSHTM Research Online to April 2015: 695

  5. Smith, O., Momber, G., Bates, R., Garwood, P., Fitch, S., Pallen, M., Gaffney, V., Allaby, RG. (2015) Sedimentary DNA from a submerged site reveals wheat in the British Isles 8000 years ago. Science 347 (6225) pp. 998-1001

    DOI lookup (Access restricted from publisher site)
    Total downloads from Research@St Andrews to April 2015: 11

  6. Hobaiter, C., & Byrne, R.W., (2014) The meanings of chimpanzee gestures Current Biology 24(14) pp. 1596-1600

    DOI lookup (Access restricted from publisher site)
    Total downloads from Research@St Andrews to April 2015: 245

  7. Darby, SC, Ewertz, M, McGale, P, Bennet, AM, Blom-Goldman, U, Bronnum, D, Correa, C, Cutter, D, Gagliardi, G, Gigante, B, Jensen, M-B, Nisbet, A, Peto, R, Rahimi, K, Taylor, C and Hall, P (2013) Risk of Ischemic Heart Disease in Women after Radiotherapy for Breast Cancer NEW ENGLAND JOURNAL OF MEDICINE, 368 (11). 987 – 998. ISSN 0028-4793

    DOI lookup (Freely available from publisher Copyright © 2013 Massachusetts Medical Society. All rights reserved.)
    Total downloads from Surrey Research Insight Open Access to April 2015: 368

  8. Otto, A, Otto, FEL, Allen, MR, Boucher, O, Church, J, Hegerl, G, Forster, PM, Gillett, NP, Gregory, J, Johnson, GC, Knutti, R, Lohmann, U, Lewis, N, Marotzke, J, Stevens, B, Myhre, G and Shindell, D (2013) Energy budget constraints on climate response. Nature Geoscience 6 (6). 415 – 416. ISSN 1752-0894

    DOI lookup (Access restricted from publisher site)
    Total downloads from White Rose Research Online to April 2015: 517

  9. Costa, Marta D, Pereira, Joana B, Pala, Maria, Fernandes, Veronica, Olivien, Anna, Achilli, Alessandro, Perego, Ugo A., Rychkov, Sergei, Naumova, Oksana, Hatina, Jiri, Woodward, Scott R., Eng, Ken Khong, Macaulay, Vincent, Carr, Martin, Soares, Pedro, Pereira, Luísa and Richards, Martin B. (2013) A substantial prehistoric European ancestry amongst Ashkenazi maternal lineages. Nature Communications 4 (2543). ISSN 2041-1723

    DOI lookup (Available OA from publisher: CC-BY)
    Total downloads from The University of Huddersfield Repository to April 2015: 104

  10. Wood, Michael J. and Douglas, Karen (2013) “What about building 7?” A social psychological study of online discussion of 9/11 conspiracy theories. Frontiers in Psychology 4 (N/A). p. 409. ISSN 1664-1078.

    DOI lookup (Available OA from publisher: CC-BY)
    Total downloads from Kent Academic Repository (KAR) to April 2015: 1244

What next for repositories and for UKCoRR?

Since 18th December 2014 the HE sector has been in thrall to the REF results, with those that did well clamouring about it and those that did less well cherry-picking the data. And clamouring about it. For the UKCoRR membership, however, REF 2014 is perhaps little more than a sideshow as we have long since been looking forward to the *next* REF when repositories, we are told, will really come of age. We built them expecting them to come, and while some did, many more stayed away, but from April the 1st 2016 even the most recalcitrant academic will need to be escorted to the repository gates the moment their paper is accepted for publication, or within 3 months at any rate. No, sorry Professor, it’s not an April Fool…

In addition to this primary requirement, there are other fundamental, related, issues most notably APC management and Research Data Management and with little more than a year to go, the Big Question is whether repository managers, as HEFCE’s foot-soldiers, have the infrastructure, resources and expertise to achieve full green Open Access in the UK – which is surely the implicit goal – and how various stakeholders – UKCoRR, Jisc, Publishers, Universities – are collaborating and responding to the considerable challenge ahead.

The UKCoRR membership now stands at over 300 members representing well over 100 institutions and organisations, the majority of which are using either EPrints or DSpace, sometimes with a CRIS (PURE, Symplectic, Converis) though often without, and with a long-tail of other software platforms. There are also different types of repository, as there are institutions, with some managing teaching and learning resources for example, e-theses or, increasingly, research data; some have sought to manage different content with a single platform (eg. Hydra) while others have opted for multiple, specialised repository instances. Some research repositories – historically a minority – are full text only whereas the majority have tended to also include bibliographic metadata, a pragmatic approach that reflects the historic difficulties encouraging academics to self archive their work. Both EPrints and DSpace are Open Source of course and some Universities run and develop in-house while others favour software as a service, outsourcing to EPrints Services for example. Each of these approaches, of course, requires specific resources and expertise.

On the UKCoRR members’, and various other software specific mailing lists, as well as at various real-life events, I cannot be the only one who has noticed a pervading uncertainty amongst those that manage and develop these, suddenly crucial, University systems, which is hardly surprising given the HEFCE requirements and the range of technology, whether in place or in development: RIOXX, CASRAI, OA Monitor, Publications Router, CORE, IRUS-UK, to mention a few.

One idea that has recently emerged from the committee is that we should, as an organisation, seek to define some sort of guidance, perhaps even a “repository specification” to help our members and their organisations to ensure that their infrastructure and advocacy is fit for purpose. There are already a wide range of relevant projects out there, notably the Jisc Pathfinder projects* – – and OAWAL (Open Access Workflows for Academic Librarians) at the University of Huddersfield – – so perhaps this is unnecessary. Please let us know what you think.

* An example of a Jisc pathfinder project exploring this area is HHuLOA OA which has sought to create a baseline of current OA activity within institutions as a way of identifying areas that require attention. Chris Awre of the project has recently disseminated a Google spreadsheet, encouraging other institutions to add their own information, in addition to the project partners – Hull, Huddersfield and Lincoln – and which is openly shared under a CC-BY licence at the link below:

See here for a blog post on the baseline –