Results of the Sherpa FACT accuracy testing – 95% accurate

UKCoRR welcomes the results of a  recent exercise – undertaken by UK librarians, repository managers and Sherpa Services – that has shown that the results produced by SHERPA/FACT (Funders & Authors Compliance Tool) have an accuracy rate of over 95%

The FACT service was developed to help researchers get a simple answer to the question “does this journal have an open access publishing policy compliant with my funder’s open access mandate?”. The FACT service – which draws its information from the SHERPA/ROMEO and SHERPA/JULIET databases – seeks to provide a yes/no answer to this question, as well as providing information about how an author can comply with a funder policy.

There had been some discussion at the SHERPA/FACT Advisory Board, raised by UKCoRR members and their institutions – as to whether the information provided by FACT was accurate.

To address this issue an exercise was undertaken – by members of UKCoRR – to manually check a statistically significant number of journal/funder combinations and then compare the information this group had found with the information provided by FACT. Where the independent reviewers arrived at a different conclusion to that provided by FACT, then that journal/publisher combination was subjected to detailed and exhaustive investigation to arrive at an evidenced answer.

At the end of this exercise, it was found that the FACT service provides correct information in over 95% of cases.

The study clearly highlighted the difficulties that even highly experienced repository staff have at deciphering publisher OA policies. Indeed, the initial testing undertaken by UKCoRR members suggested that FACT was only accurate on 57% of occasions. When these journal/funder combinations were investigated further, however, close examination of the often complex conditions and the interactions between different statements and policies showed that FACT was correct in almost all of the cases.

The SHERPA/FACT team as well as the SHERPA/FACT Advisory Group would like to extend their thanks to the UKCoRR Members who took part in this checking process for their time commitment as well as their extensive knowledge of this area of work. This exercise has proved that the SHERPA/FACT service can be relied upon as a source of advice for UK researchers. UKCoRR also encourages it’s members to continue to communicate with SHERPA/FACT where discrepancies are found to continue to improve the quality of the information SHERPA/FACT relies upon.

Issues of interpretation and the interaction between the various policies have been seen to be the key to the discrepancy between our manually checked results and Sherpa’s findings. There is further work to be done here and UKCoRR looks forward to continuing to work with the SHERPA/FACT Advisory Group to develop increased clarity in this area.

To see the full data and study methodology, visit Figshare.

The study was commissioned by the SHERPA/FACT Advisory Board – which includes representatives from UKCoRR, Jisc, Wellcome Trust, Research Councils UK (RCUK), CRC Nottingham, Association of Learned & Professional Society Publishers (ALPSP), Higher Education Funding Council for England (HEFCE), Publishers Association and SCONUL.

A blog post from Jisc on this project is available as well as a press release on the project.