Preface: We asked Anita Bandrowski from SciScore about reliable methods reporting and learned about tools developed to help researchers write complete and rigorous methods sections and increase scientific reproducibility. While SciScore can help us fill out the MDAR reports and STAR tables required by specific journals, it also gives a framework for checking that things like oligos, statistics, and code are listed in a manuscript before someone leaves the lab. This interview is a part of our ongoing series talking to people that are using innovation to help support your research.
Can you tell us about yourself and what led you to create SciScore?
I have worked both at the bench and in informatics so in approaching scientific topics I tend to have the mindset of a bench scientist when approaching informatics problems, which can be beneficial in some cases.
SciScore really came out of a frustration of many bench scientists that have tried to find antibodies and other resources and simply could not do it. We first started the RRID project, which asks authors to look up persistent identifiers for antibodies and other research resources used in the paper. The problem with implementation is that someone, or we hope something, must remind authors to add the information to their paper. The tool was intended to remind authors before they published that they had omitted something before that something is lost to the “postdoc leaving the lab problem”.
As a researcher, I have struggled with trying to reproduce methods from research papers. How does SciScore help the community with this?
SciScore was intended to be ‘gamed’, i.e., it gives authors who remember to address various things like properly naming their mouse or remembering to put in inclusion and exclusion criteria a higher numerical score. Since SciScore is used during the publishing process, it reminds authors if they have forgotten something. It is those little things (can’t find an antibody or the right mouse) that make it very difficult to replicate a study, but are easy to fix with just a little more information from the author. This does not guarantee reproducibility of a study, but it definitely helps to get us in that direction.
Can you tell us about the different reports from SciScore, and when we researchers would need to know if our manuscripts have a good score?
So SciScore produces 3 reports: the author report (most authors get this from their journals), the MDAR report, and the STAR table. The information and the score is accessible from the author report, as this is intended to be as interactive as possible. The report shows the sentence that authors used and the item extracted from the sentence, in most cases. This is also the only report with a score, and that is intended to be an overall measure of how many items were found vs the number that were expected by the tool. The report is intended for iteration so as the paper addresses omitted things, the score improves.
The MDAR report is intended to be submitted to a journal that requires the report. It is based on the same core information, but it is not scored and it is a little more difficult to check and verify correctness. An example of a journal that requires the MDAR is AAAS’s journal Science. Here at acceptance of the paper there is a requirement for the MDAR to be filled in. SciScore can produce the filled out checklist immediately and it is based on the paper itself, something that the raw checklist does not guarantee since it can be filled out without augmenting the paper itself.
The STAR table is a csv file that only includes the resources table. It is intended to be used by authors who would like to look up some of their RRIDs automatically. It generally does not include all resources for the full STAR table to be just copied and pasted, but it can help authors get started with a file that is easy to edit.
How can SciScore help researchers learn and follow best practices for reliable methods reporting in their manuscripts?
When we read papers entitled “ARRIVE has not arrived” we realize that checklists are hard for people to follow. Usually scientists write their papers with flow and clarity in mind and not a checklist, so SciScore is a good assistant. It takes the methods section and quickly does a check for the most common checklist items. Most of the time, especially if they have written their papers a certain way before, authors forget to put in whether or not they had blinded the study or whether they had checked group size using the power calculation. If they are reminded, they can at least tell readers what they did, improving transparency.
Do you think scientific publishing is invested in moving towards requiring more reliable methods reporting?
There is an appetite for it, it seems. However, before tools like SciScore, checking rigor items was immensely costly for journals and so only journals like Nature could really implement a checklist that was verified by expensive human beings. Other journals asked authors to fill some information into checklists or reviewers to do these checks. These are good efforts, but they tend to fall a little flat, especially if checklists are not backed up by anything.
Imagine, if there is a checkbox “did you include all necessary RRIDs into your manuscript?” and you would need to check the box that you did but there is nothing that checks that you actually did it. Some people will simply check a box to just move to the next step in submission. If a tool was able to verify this check, there would be more incentive for authors to improve their manuscript.
I saw in the RDI reporting that journals can vary a lot in their scores and the number of materials reported—are there particular fields or specialties where SciScore and reliable methods reporting are particularly important?
The Rigor and Transparency index is different by field, and largely follows fields which adhere to guidelines more closely. For example medical journals generally score higher, because authors generally follow CONSORT or other guidelines fairly closely (how subjects are randomized, and when they drop out of the study is closely monitored by the community because that can signal poor outcomes affecting patient’s lives). Medical studies need to be pre-registered, whereas these standards are less commonly followed in basic biology journals resulting in lower scores, and almost unheard of in chemistry journals.
The cancer literature is one place where SciScore is, I think, perhaps the most important. While not all rigor items are important for the cancer literature, the inclusion of RRIDs largely prevents the use of contaminated and misidentified cell lines, thus SciScore reminds authors to add RRIDs and authenticate cell lines, cleaning up places where there are reagent-specific reasons to remove data.
Can you share what your vision for the future of reproducible research is?
You know I like chess and one thing I like about it is the automation of certain moves into a strategy. I remember reading that chess grand masters do not have better memory for the game than anyone else, but they think in larger multi-move chunks, and not individual moves.
I envision that in the future, there is a set of bots that can help humans check on things that are boring, like checking individual reagent information and authentication. These rudimentary checks should put us on a surer footing, methodologically freeing humans to do the things that humans are uniquely good at, such as thinking about the implications of the study.
Perhaps this is thinking too far ahead, but I also envision a time where a study could be replicated by a robot from instructions written by another robot in a journal. I actually think that replicating studies is kind of boring, but if we can get a system in place to do it automatically we might be able to take on bigger tasks, like the overall strategy in chess knowing that the individual moves will be completed accurately.
Acknowledgment
We would like to thank Anita for sharing her insight!
