Software tools for systematic review literature screening and data extraction: Qualitative user experiences from succinct formal tests
Article Sidebar
Main Article Content
Abstract
Systematic reviews (SRs) are an important tool in implementing the 3Rs in preclinical research. With the ever-increasing amount of scientific literature, SRs require increasing time-investments. Thus, using the most efficient review tools is essential. Most available tools aid the screening process, tools for data-extraction and / or multiple review phases are relatively scarce. Using a single platform for all review phases allows for auto-transfer of references from one phase to the next, which enables work on multiple phases at the same time. We performed succinct formal tests of four multiphase review tools that are free or relatively affordable: Covidence, Eppi, SRDR+ and SYRF. Our tests comprised full-text screening, sham data extraction and discrepancy resolution in the context of parts of a systematic review. Screening was performed as per protocol. Sham data extraction comprised free text, numerical and categorial data. Both reviewers kept a log of their experiences with the platforms throughout. These logs were qualitatively summarized and supplemented with further user experiences. We show value of all tested tools in the SR process. Which tool is optimal depends on multiple factors, comprising previous experience with the tool, but also review type, review questions and review team member enthusiasm.
Plain language summary
Systematic reviews (SRs) are reliable summaries of tests that have been done in the past. They are an important tool in improving animal welfare and in decreasing the use of animals in research. However, because new studies are published all the time, summarizing them reliably takes more and more time. People doing an SR can use different tools to help them, which can save time. Many tools are available, and we did a brief study to compare some of them. We chose four tools that are free or low-cost: Covidence, Eppi, SRDR+ and SYRF. We tested how they would work in different steps of an SR. During testing, two reviewers wrote down all their experiences. We summarize the results in this paper. All four tested tools can help reviewers to work efficiently. We advise which tool can help the most in different settings.
Article Details
This work is licensed under a Creative Commons Attribution 4.0 International License.
Articles are distributed under the terms of the Creative Commons Attribution 4.0 International license (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution and reproduction in any medium, provided the original work is appropriately cited (CC-BY). Copyright on any article in ALTEX is retained by the author(s).
Beller, E., Clark, J., Tsafnat, G. et al. (2018). Making progress with the automation of systematic reviews: Principles of the international collaboration for the automation of systematic reviews (icasr). Syst Rev 7, 77. doi:10.1186/s13643-018-0740-7
Bramer, W. M., Milic, J. and Mast, F. (2017). Reviewing retrieved references for inclusion in systematic reviews using endnote. J Med Libr Assoc 105, 84-87. doi:10.5195/jmla.2017.111
Cleo, G., Scott, A. M., Islam, F. et al. (2019). Usability and acceptability of four systematic review automation software packages: A mixed method design. Syst Rev 8, 145. doi:10.1186/s13643-019-1069-6
Cowie, K., Rahmatullah, A., Hardy, N. et al. (2022). Web-based software tools for systematic literature review in medicine: Systematic search and feature analysis. JMIR Med Inform 10, e33219. doi:10.2196/33219
Gates, A., Guitard, S., Pillay, J. et al. (2019). Performance and usability of machine learning for screening in systematic reviews: A comparative evaluation of three tools. Syst Rev 8, 278. doi:10.1186/s13643-019-1222-2
Harrison, H., Griffin, S. J., Kuhn, I. et al. (2020). Software tools to support title and abstract screening for systematic reviews in healthcare: An evaluation. BMC Med Res Methodol 20, 7. doi:10.1186/s12874-020-0897-3
Leenaars, C., Tsaioun, K., Stafleu, F. et al. (2021). Reviewing the animal literature: How to describe and choose between different types of literature reviews. Lab Anim 55, 129-141. doi:10.1177/0023677220968599
Leenaars, C., Hager, C., Stafleu, F. et al. (2023). A systematic review of the effect of cystic fibrosis treatments on the nasal potential difference test in animals and humans. Diagnostics (Basel) 13, doi:10.3390/diagnostics13193098
Leenaars, C. H. C., Stafleu, F. R., Hager, C. et al. (2024a). A case study of the informative value of risk of bias and reporting quality assessments for systematic reviews. Syst Rev 13, 230. doi:10.1186/s13643-024-02650-w
Leenaars, C. H. C., Stafleu, F. R., Hager, C. et al. (2024b). A systematic review of animal and human data comparing the nasal potential difference test between cystic fibrosis and control. Sci Rep 14, 9664. doi:10.1038/s41598-024-60389-9
O'Connor, A. M., Tsafnat, G., Thomas, J. et al. (2019). A question of trust: Can we build an evidence base to gain trust in systematic review automation technologies? Syst Rev 8, 143. doi:10.1186/s13643-019-1062-0
Page, M. J., McKenzie, J. E., Bossuyt, P. M. et al. (2021). The prisma 2020 statement: An updated guideline for reporting systematic reviews. BMJ 372, n71. doi:10.1136/bmj.n71
Russell, W. M. S. and Burch, R. L. (1959). The principles of humane experimental technique. Vol. London, UK.: Methuen.
Van der Mierden, S., Tsaioun, K., Bleich, A. et al. (2019). Software tools for literature screening in systematic reviews in biomedical research. ALTEX 36, 508-517. doi:10.14573/altex.1902131
Van der Mierden, S., Spineli, L. M., Talbot, S. R. et al. (2021). Extracting data from graphs: A case-study on animal research with implications for meta-analyses. Res Synth Methods 12, 701-710. doi:10.1002/jrsm.1481
Wang, Z., Nayfeh, T., Tetzlaff, J. et al. (2020). Error rates of human reviewers during abstract screening in systematic reviews. PLoS One 15, e0227742. doi:10.1371/journal.pone.0227742