Network effects: on alternative metrics

“No one can read everything. We rely on filters to make sense of the scholarly literature, but the narrow, traditional filters are being swamped. However, the growth of new, online scholarly tools allows us to make new filters; these altmetrics reflect the broad, rapid impact of scholarship in this burgeoning ecosystem. We call for more tools and research based on altmetrics.”

altmetrics: a manifesto

Research metrics are nothing if not controversial, none more so than the “alternative” variety currently in vogue from Plum Analytics, ImpactStory and the donut people themselves altmetric.com. For the most part, the twitterati are buying into them in a big way, not to mention Big Publishers and Open Access advocates, though some influential voices are dismissive. At best. See this title from David Colquhoun and Andrew Plested “Why you should ignore altmetrics and other bibliometric nightmares“. Or even read it – it’s hard to deny that most folk who retweet the monster-altmetric articles probably don’t (personally I eschew the RT without having read an article, whether academic or pop-cultural trivia. Maybe that’s just me*)

* I’ll admit occasionally to having retweeted *before* reading but only if I trust the source.

Though a mere bibliometric tinkerer myself, I rather take umbrage on behalf of professional bibliometricians everywhere who are somewhat maligned in this article: “The mistake made by all bibliometricians is that they fail to consider the content of papers, because they have no desire to understand research.” This arguably needs a citation and at least one cohort of bibliometricians to take an IQ test to establish if bibliometrics really are “for people who aren’t prepared to take the time (or lack the mental capacity) to evaluate research by reading about it.” I can’t necessarily back up with a citeable source myself but I’m willing to bet that the very first thing one learns as a bibliometrician is the strengths and, often profound, weaknesses of the various metrics, whether JIF, h-index or alternative. To dismiss their value entirely seems to rather throw the baby out with the bathwater.

Not that I don’t have reservations – the colour coded donut of altmetric.com surrounding a number arguably lacks clarity and I’m not entirely certain what the number actually refers to, though it seems to correlate vaguely with tweets (coloured blue in the donut) which is invariably the highest “metric” – and arguably the least valuable – there is presumably some weighting towards potentially more valuable, less ephemeral activity on blogs (yellow) and in mainstream news outlets (dark red). I’m not convinced Mendeley and CiteULike necessarily reflect representative networks across all disciplines (dark red and light blue respectively) or Facebook for that matter (dark blue)…however, this is surely the point, the network, and while altmetrics do perhaps pander to a human propensity to quantify with raw numbers – though no more than traditional citations – their main strength is undoubtedly to visualise and leverage the network; to connect with peers online, or with interested laypersons, to foster community and promote research activity and impact beyond the walls of the ivory tower.

N.B. I’ve just picked up this article from Twitter – Numbers behind Numbers: The Altmetric Score and Sources Explained…but haven’t read it yet!

Donuts can be embedded easily using code provided at http://api.altmetric.com/embeds.html and the API can be accessed free of charge, with various paid options for more sophisticated functionality, though with a little knowhow you can derive useful data without a subscription to the premium service. There is also a plugin available from the EPrints Bazaar

IRUS_april

Below are the top 10 articles by altmetric for repository downloads in April 2015 as recorded by IRUS-UK and derived from 84 UK repositories and a total of more than 87,000 unique items. Data is presented with a link to the primary repository and highlights numbers of total downloads* from this, and any other repositories where the paper is archived (*i.e. all downloads from each repository, not just April). There is a DOI lookup link and a note whether the article is available to download from the publisher’s site (N.B. only 3 of these 10 are restricted access with 5 under CC-By and 2 freely accessible albeit © All Rights Reserved).

No further analysis is offered other than to say that there does not appear to be any obvious correlation between altmetrics and repository downloads – number 1 is obviously related to a major story in the media about a certain King found in a Leicester carpark – but, it bears reiterating, that is not necessarily the point; rather repositories are an embedded, institutionally managed element of researchers’ online network with tools like IRUS and altmetrics enabling data to be visualised and leveraged across an increasingly coherent infrastructure.

N.B. Information on how this data was derived using the export function from IRUS-UK and a Google Sheet will follow in a subsequent post.

  1. King, T.E., Fortes, G.G., Balaresque, P., Thomas, M.G., Balding, D., Delser, P.M., Neumann, R., Parson, W., Knapp, M., Walsh, S., Tonasso, L., Holt, J., Kayser, M., Appleby, J., Forster, P., Ekserdjian, D., Hofreiter, M., Schürer, K. (2014) Identification of the remains of King Richard III. Nature Communications, 5, art. no. 5631

    DOI lookup (Available OA from publisher: CC-BY)
    Total downloads from Leicester Research Archive to April 2015:
    132

  2. Sumner, Petroc, Vivian-Griffiths, Solveiga, Boivin, Jacky, Williams, Andrew James, Venetis, C. A., Davies, Aimee, Ogden, Jack, Whelan, Leanne, Hughes, Bethan, Dalton, Bethan, Boy, Frederic and Chambers, Christopher D. 2014. The association between exaggeration in health related science news and academic press releases: retrospective observational study. BMJ 349, g7015

    DOI lookup (Available OA from publisher: CC-BY)
    Total downloads from Online Research @ Cardiff (ORCA) to April 2015: 42

  3. ATLAS collaboration (2012) Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC. Physics Letters B, Volume 716 (Number 1). pp. 1-29. ISSN 0370-2693

    DOI lookup (Available OA from publisher: CC-BY)
    Total downloads from
    Lancaster EPrints to April 2015: 29
    Total downloads from
    WRAP: Warwick Research Archive Portal to April 2015: 16

  4. RTS,S Clinical Trials Partnership; (2011) First results of phase 3 trial of RTS,S/AS01 malaria vaccine in African children. The New England journal of medicine, 365 (20). pp. 1863-75. ISSN 0028-4793

    DOI lookup (Freely available from publisher: © Massachusetts Medical Society) 
    Total downloads from LSHTM Research Online to April 2015: 695

  5. Smith, O., Momber, G., Bates, R., Garwood, P., Fitch, S., Pallen, M., Gaffney, V., Allaby, RG. (2015) Sedimentary DNA from a submerged site reveals wheat in the British Isles 8000 years ago. Science 347 (6225) pp. 998-1001

    DOI lookup (Access restricted from publisher site)
    Total downloads from Research@St Andrews to April 2015: 11

  6. Hobaiter, C., & Byrne, R.W., (2014) The meanings of chimpanzee gestures Current Biology 24(14) pp. 1596-1600

    DOI lookup (Access restricted from publisher site)
    Total downloads from Research@St Andrews to April 2015: 245

  7. Darby, SC, Ewertz, M, McGale, P, Bennet, AM, Blom-Goldman, U, Bronnum, D, Correa, C, Cutter, D, Gagliardi, G, Gigante, B, Jensen, M-B, Nisbet, A, Peto, R, Rahimi, K, Taylor, C and Hall, P (2013) Risk of Ischemic Heart Disease in Women after Radiotherapy for Breast Cancer NEW ENGLAND JOURNAL OF MEDICINE, 368 (11). 987 – 998. ISSN 0028-4793

    DOI lookup (Freely available from publisher Copyright © 2013 Massachusetts Medical Society. All rights reserved.)
    Total downloads from Surrey Research Insight Open Access to April 2015: 368

  8. Otto, A, Otto, FEL, Allen, MR, Boucher, O, Church, J, Hegerl, G, Forster, PM, Gillett, NP, Gregory, J, Johnson, GC, Knutti, R, Lohmann, U, Lewis, N, Marotzke, J, Stevens, B, Myhre, G and Shindell, D (2013) Energy budget constraints on climate response. Nature Geoscience 6 (6). 415 – 416. ISSN 1752-0894

    DOI lookup (Access restricted from publisher site)
    Total downloads from White Rose Research Online to April 2015: 517

  9. Costa, Marta D, Pereira, Joana B, Pala, Maria, Fernandes, Veronica, Olivien, Anna, Achilli, Alessandro, Perego, Ugo A., Rychkov, Sergei, Naumova, Oksana, Hatina, Jiri, Woodward, Scott R., Eng, Ken Khong, Macaulay, Vincent, Carr, Martin, Soares, Pedro, Pereira, Luísa and Richards, Martin B. (2013) A substantial prehistoric European ancestry amongst Ashkenazi maternal lineages. Nature Communications 4 (2543). ISSN 2041-1723

    DOI lookup (Available OA from publisher: CC-BY)
    Total downloads from The University of Huddersfield Repository to April 2015: 104

  10. Wood, Michael J. and Douglas, Karen (2013) “What about building 7?” A social psychological study of online discussion of 9/11 conspiracy theories. Frontiers in Psychology 4 (N/A). p. 409. ISSN 1664-1078.

    DOI lookup (Available OA from publisher: CC-BY)
    Total downloads from Kent Academic Repository (KAR) to April 2015: 1244

What next for repositories and for UKCoRR?

Since 18th December 2014 the HE sector has been in thrall to the REF results, with those that did well clamouring about it and those that did less well cherry-picking the data. And clamouring about it. For the UKCoRR membership, however, REF 2014 is perhaps little more than a sideshow as we have long since been looking forward to the *next* REF when repositories, we are told, will really come of age. We built them expecting them to come, and while some did, many more stayed away, but from April the 1st 2016 even the most recalcitrant academic will need to be escorted to the repository gates the moment their paper is accepted for publication, or within 3 months at any rate. No, sorry Professor, it’s not an April Fool…

In addition to this primary requirement, there are other fundamental, related, issues most notably APC management and Research Data Management and with little more than a year to go, the Big Question is whether repository managers, as HEFCE’s foot-soldiers, have the infrastructure, resources and expertise to achieve full green Open Access in the UK – which is surely the implicit goal – and how various stakeholders – UKCoRR, Jisc, Publishers, Universities – are collaborating and responding to the considerable challenge ahead.

The UKCoRR membership now stands at over 300 members representing well over 100 institutions and organisations, the majority of which are using either EPrints or DSpace, sometimes with a CRIS (PURE, Symplectic, Converis) though often without, and with a long-tail of other software platforms. There are also different types of repository, as there are institutions, with some managing teaching and learning resources for example, e-theses or, increasingly, research data; some have sought to manage different content with a single platform (eg. Hydra) while others have opted for multiple, specialised repository instances. Some research repositories – historically a minority – are full text only whereas the majority have tended to also include bibliographic metadata, a pragmatic approach that reflects the historic difficulties encouraging academics to self archive their work. Both EPrints and DSpace are Open Source of course and some Universities run and develop in-house while others favour software as a service, outsourcing to EPrints Services for example. Each of these approaches, of course, requires specific resources and expertise.

On the UKCoRR members’, and various other software specific mailing lists, as well as at various real-life events, I cannot be the only one who has noticed a pervading uncertainty amongst those that manage and develop these, suddenly crucial, University systems, which is hardly surprising given the HEFCE requirements and the range of technology, whether in place or in development: RIOXX, CASRAI, OA Monitor, Publications Router, CORE, IRUS-UK, to mention a few.

One idea that has recently emerged from the committee is that we should, as an organisation, seek to define some sort of guidance, perhaps even a “repository specification” to help our members and their organisations to ensure that their infrastructure and advocacy is fit for purpose. There are already a wide range of relevant projects out there, notably the Jisc Pathfinder projects* – http://openaccess.jiscinvolve.org/wp/pathfinder-projects/ – and OAWAL (Open Access Workflows for Academic Librarians) at the University of Huddersfield – https://library3.hud.ac.uk/blogs/oawal/ – so perhaps this is unnecessary. Please let us know what you think.

* An example of a Jisc pathfinder project exploring this area is HHuLOA OA which has sought to create a baseline of current OA activity within institutions as a way of identifying areas that require attention. Chris Awre of the project has recently disseminated a Google spreadsheet, encouraging other institutions to add their own information, in addition to the project partners – Hull, Huddersfield and Lincoln – and which is openly shared under a CC-BY licence at the link below:

https://docs.google.com/spreadsheets/d/1MN7Qw_wlU2LMGnlcmjzgufJZuhzL6F_ay4lhcQEGKv8/edit#gid=0

See here for a blog post on the baseline – https://library3.hud.ac.uk/blogs/hhuloa/2015/02/05/open-access-baseline-activity-tool/

6th International Open Access Week (20 – 26 October 2014)

In 2008 I was a second year doctoral student at Simmons College, Boston, MA, USA when my mentors Robin Peek and Peter Suber asked me to prepare an event at the College to celebrate the first International Open Access Day (October 14th 2008). Back then the open access movement (OA) was less well established than today and OA advocates, like myself, often faced considerable resistance from scholars. Seven years on and now that, rather than a single day, OA week is an annual and international event, I am thrilled to see that the movement has gained such momentum and that events are organised all over the world.

Almost a month ago I emailed the UKCoRR listserve membership asking about their plans for this year’s OA week. I received plenty of replies, so a big thank you to those who responded!

The main focus of this year’s events is no surprise, with the new HEFCE Open Access Policy and its implications on the post-2014 Research Excellence Framework being presented at the vast majority, targeting both compliance and deposit requirements. Other topics include general presentations on open access and the various routes to OA (e.g. green vs gold), ORCiD ids and copyright. Some institutions have also arranged subject specific presentations, i.e. humanities and sciences, with presentations on how open access specifically relates to these fields. Jisc will celebrate the launch of a wonderful project, the Open Access Button, which enables users to record and ‘map’ outputs where access is restricted by a paywall, and also includes technology to source an OA version of the article (e.g. from a repository).  In addition, events this year explore Research Data Management (RDM) practice, since more and more funders are mandating not only the open accessibility of research output, but also of the data that accompanies these outputs.

There are a wide variety of events, with UK HEIs running face to face events, online webinars, formal presentations and informal discussions, some of which are addressed solely to internal delegates or to both internal and external delegates.

Here is a list of events that were announced on the UKCoRR listserve (in alphabetical order):

I am sure that other institutions are also planning to run their own events and UKCoRR would love to know more about them. Feel free to add your event further down in the “Comments” and please include links to presentations.

Happy Open Access Week 2014!
May this year our repositories grow in full-text deposits and flourish!
Enjoy!

 

UKCoRR Response to HEFCE’s Open Access Policy for the Post-2014 REF

Following their consultation in the summer of last year[1] HEFCE have released their policy on open access in the post-2014 REF process.  This is the third open access policy from a major UK funder in as many years and there are lot of reasons to be cheerful.  HEFCE’s policy as published this morning is a genuine, cost effective route to widening the access to the UK’s research outputs.

Firstly I would like to commend HEFCE for their acknowledgement of the work done by the UK repository community, both institutionally based and subject based something that has been disappointingly lacking in the other policies of its type.  The UK has a (still-) growing and passionate repository community who are doing great work which has been misunderstood and poorly valued by the Finch Report in particular.  The new policy from HEFCE is a chance to stand-up and demonstrate the value of our services to the academic community and to the other research funders as well.

I would also like to acknowledge the commitment of HEFCE to work with the repository community to ensure that all of our systems are ready to comply with this policy by the time it takes effect, regardless of their shape and set up.  There are a number of issues and process questions that the policy leaves unresolved at the moment, these we would urge HEFCE to make clear as soon as possible.  This issues have the potential to cause a number of resourcing issues for HEIs in terms of tracking and ensuring compliance.  We look forward to discussing these process issues further with HEFCE.

HEFCE’s policy also takes a pragmatic view on the issues of licensing and exceptions.  There is a strong awareness of the complex nature of academic publishing throughout the policy.  There is also a real sense that HEFCE is trying to take into account every part of the UK academic community in a way that accepts the distinct needs of the different disciplines.  We also welcome the commitment implicit in HEFCE’s policy for researchers continuing to choose the “most appropriate” venue of publication for their outputs even if it means their options for compliance are reduced.

The suggestion of the Creative Commons Attribution-Non-Commercial-No-Derivatives (CC-BY-NC-ND) license and the fact that requirements to allow text-mining are now missing from the policy as well as the extensive list of allowed exceptions[2] make the policy practical but do not go as far as many would have liked.  However they will allow institutions to meet the requirements comfortably.  We need to remember that these are minimal requirements, we are always free to strive for more and HEFCE have stated that they will acknowledge the efforts of those who do[3].

This is a policy routed in the belief that the route to open access is a long term one and will only be achieved incrementally.  HEFCE’s policy coupled with the policies of bodies such as RCUK, the Wellcome Trust, Horizon 2020 and others are part of the continuum of open access and unless the underlying business models that drive this sector change we won’t ever get true or ‘libre’ open access as it is just not financially practical.  We in UKCoRR have the skills, knowledge and passion to make this work and I look forward to working with HEFCE, Jisc and our researchers to do just that.


[1] UKCoRR’s Response to the HEFCE consultation has been published on this site along with our responses to other similar consultation documents.

[2] A full list of the permitted exceptions in their categories has been extracted from the HEFCE policy document by UKCoRR for the use of their members and others.