How evidence-based methodologies can help identify and reduce uncertainty in chemical risk assessment

Main Article Content

Sebastian Hoffmann
Paul Whaley
Katya Tsaioun


Evidence-based methodology, in particular systematic review, is increasingly being applied in environmental, public, and occupational health to increase the transparency, comprehensiveness, and objectivity of the processes by which existing evidence is gathered, assessed, and synthesized in answering research questions. This development is also changing risk assessment practices and will impact the assessment of uncertainties in the evidence for risks to human health that are posed by exposure to chemicals. The potential of evidence-based methodology for characterizing uncertainties in risk assessment has been widely recognized, while its contribution to uncertainty reduction is yet to be fully elucidated. We therefore present some key aspects of the evidence-based approach to risk assessment, showing how they can contribute to the identification and the assessment of uncertainties. We focus on the pre-specification of an assessment method­ology in a protocol, comprehensive search strategies, study selection using predefined eligibility criteria, critical appraisal of individual studies, and an evidence integration and uncertainty characterization process based on certainty of evi­dence frameworks that are well-established in health care research. We also provide examples of uncertainty in risk assessment and discuss how evidence-based methodology could address those. This perspective, which neither claims to be comprehensive nor complete, is intended to stimulate discussion of the topic and to motivate detailed exploration of how evidence-based methodology contributes to characterization of uncertainties, and how it will lead to uncertainty reduction in the conduct of health risk assessment.

Article Details

How to Cite
Hoffmann, S., Whaley, P. and Tsaioun, K. (2022) “How evidence-based methodologies can help identify and reduce uncertainty in chemical risk assessment”, ALTEX - Alternatives to animal experimentation, 39(2), pp. 175–182. doi: 10.14573/altex.2201131.
Food for Thought ...

Beck, N. B., Becker, R. A., Erraguntla, N. et al. (2016). Approaches for describing and communicating overall uncertainty in toxicity characterizations: U.S. Environmental Protection Agency’s integrated risk information system (IRIS) as a case study. Environ Int 89-90, 110-128. doi:10.1016/j.envint.2015.12.031

Benbrook, C. M. (2019). How did the US EPA and IARC reach diametrically opposed conclusions on the genotoxicity of glyphosate-based herbicides? Environ Sci Eur 31, 2. doi:10.1186/s12302-018-0184-7

Berkman, N. D., Santaguida, P. L., Viswanathan, M. et al. (2014). The Empirical Evidence of Bias in Trials Measuring Treatment Differences.

Bero, L., Chartres, N., Diong, J. et al. (2018). The risk of bias in observational studies of exposures (ROBINS-E) tool: Concerns arising from application to observational studies of exposures. Syst Rev 7, 242. doi:10.1186/s13643-018-0915-2

Bhat, V. S., Meek, M. E. B., Valcke, M. et al. (2017). Evolution of chemical-specific adjustment factors (CSAF) based on recent international experience; increasing utility and facilitating regulatory acceptance. Crit Rev Toxicol 47, 729-749. doi:10.1080/10408444.2017.1303818

Cano-Sancho, G., Ploteau, S., Matta, K. et al. (2019). Human epidemiological evidence about the associations between exposure to organochlorine chemicals and endometriosis: Systematic review and meta-analysis. Environ Int 123, 209-223. doi:doi:10.1016/j.envint.2018.11.065

Chvátalová, V. (2019). A critical evaluation of EFSA’s environmental risk assessment of genetically modified maize MON810 for honeybees and earthworms. Environ Sci Eur 31, 52. doi:10.1186/s12302-019-0238-5

Cooper, G. S., Lunn, R. M., Agerstrand, M. et al. (2016). Study sensitivity: Evaluating the ability to detect effects in systematic reviews of chemical exposures. Environ Int 92-93, 605-610. doi:10.1016/j.envint.2016.03.017

Dankovic, D. A., Naumann, B. D., Maier, A. et al. (2015). The Scientific Basis of Uncertainty Factors Used in Setting Occupational Exposure Limits. J Occup Environ Hyg 12, Suppl 1, S55-S68. doi:10.1080/15459624.2015.1060325

De Vries, R. B. M., Angrish, M., Browne, P. et al. (2021). Applying evidence-based methods to the development and use of adverse outcome pathways. ALTEX 38, 336-347. doi:10.14573/altex.2101211

Deveau, M., Chen, C.-P., Johanson, G. et al. (2015). The global landscape of occupational exposure limits – Implementation of harmonization principles to guide limit selection. J Occup Environ Hyg 12, Suppl 1, S127-144. doi:10.1080/15459624.2015.1060327

Dorman, D. C., Chiu, W., Hales, B. F. et al. (2018). Systematic reviews and meta-analyses of human and animal evidence of prenatal diethylhexyl phthalate exposure and changes in male anogenital distance. J Toxicol Environ Health B Crit Rev 21, 207-226. doi:10.1080/10937404.2018.1505354

Eddy, D. M. (2005). Evidence-based medicine: A unified approach. Health Aff (Millwood) 24, 9-17. doi:10.1377/hlthaff.24.1.9

EFSA (2010). Application of systematic review methodology to food and feed safety assessments to support decision making. EFSA J 8, 1637. doi:10.2903/j.efsa.2010.1637

EFSA (2015). Principles and process for dealing with data and evidence in scientific assessments. EFSA J 13, 4121. doi:10.2903/j.efsa.2015.4121

EFSA, Gundert-Remy, U., Bodin, J. et al. (2017). Bisphenol A (BPA) hazard assessment protocol. EFSA Supporting Publication 14, 1354E. doi:10.2903/sp.efsa.2017.EN-1354

EFSA Scientific Committee (2018a). Guidance on uncertainty analysis in scientific assessments. EFSA J 16, e05123. doi:10.2903/j.efsa.2018.5123

EFSA Scientific Committee (2018b). The principles and methods behind EFSA’s guidance on uncertainty analysis in scientific assessment. EFSA J 16, e05122. doi:10.2903/j.efsa.2018.5122

Gusenbauer, M. and Haddaway, N. R. (2021). What every researcher should know about searching – Clarified concepts, search advice, and an agenda to improve finding in academia. Res Synth Methods 12, 136-147. doi:10.1002/jrsm.1457

Guyatt, G. H., Oxman, A. D., Vist, G. E. et al. (2008). GRADE: An emerging consensus on rating quality of evidence and strength of recommendations. BMJ 336, 924-926. doi:10.1136/bmj.39489.470347.AD

Hassauer, C. and Roosen, J. (2020). Toward a conceptual framework for food safety criteria: Analyzing evidence practices using the case of plant protection products. Saf Sci 127, 104683. doi:10.1016/j.ssci.2020.104683

Hoffmann, S., De Vries, R. B. M., Stephens, M. L. et al. (2017). A primer on systematic reviews in toxicology. Arch Toxicol 9, 2551-2575. doi:10.1007/s00204-017-1980-3

Howard, B. E., Phillips, J., Tandon, A. et al. (2020). SWIFT-active screener: Accelerated document screening through active learning and integrated recall estimation. Environ Int 138, 105623. doi:10.1016/j.envint.2020.105623

Kabat, G. C. (2019). Rapid response to: Probable carcinogenicity of glyphosate. BMJ 365, l1613. doi:10.1136/bmj.l1613

Kass, G. E. N. and Lodi, F. (2020). Letter to the editor regarding the article ‘EFSA’s toxicological assessment of aspartame: Was it even-handedly trying to identify possible unreliable positives and unreliable negatives?’ Arch Public Health 78, 14. doi:10.1186/s13690-020-0395-4

Kogevinas, M. (2019). Probable carcinogenicity of glyphosate. BMJ 365, l1613. doi:10.1136/bmj.l1613

Koustas, E., Lam, J., Sutton, P. et al. (2014). The navigation guide – Evidence-based medicine meets environmental health: Systematic review of nonhuman evidence for PFOA effects on fetal growth. Environ Health Perspect 122, 1015-1027. doi:doi:10.1289/ehp.1307177

Lam, J., Koustas, E., Sutton, P. et al. (2014). The navigation guide – Evidence-based medicine meets environmental health: Integration of animal and human evidence for PFOA effects on fetal growth. Environ Health Perspect 122, 1040-1051. doi:10.1289/ehp.1307923

Li, J., Pega, F., Ujita, Y. et al. (2020). The effect of exposure to long working hours on ischaemic heart disease: A systematic review and meta-analysis from the WHO/ILO joint estimates of the work-related burden of disease and injury. Environ Int 142, 105739. doi:10.1016/j.envint.2020.105739

Lynch, H. N., Goodman, J. E., Tabony, J. A. et al. (2016). Systematic comparison of study quality criteria. Regul Toxicol Pharmacol 76, 187-198. doi:10.1016/j.yrtph.2015.12.017

Mahood, Q., Van Eerd, D. and Irvin, E. (2014). Searching for grey literature for systematic reviews: Challenges and benefits. Res Synth Methods 5, 221-234. doi:10.1002/jrsm.1106

Marshall, I. J. and Wallace, B. C. (2019). Toward systematic review automation: a practical guide to using machine learning tools in research synthesis. Syst Rev 8, 163. doi:10.1186/s13643-019-1074-9

Martin, P., Bladier, C., Meek, B. et al. (2018). Weight of evidence for hazard identification: A critical review of the literature. Environ Health Perspect 126, 076001. doi:10.1289/EHP3067

Matta, K., Ploteau, S., Coumoul, X. et al. (2019). Associations between exposure to organochlorine chemicals and endometriosis in experimental studies: A systematic review protocol. Environ Int 124, 400-407. doi:10.1016/j.envint.2018.12.063

Meek, M. E., Boobis, A., Cote, I. et al. (2014). New developments in the evolution and application of the WHO/IPCS framework on mode of action/species concordance analysis. J Appl Toxicol 34, 1-18. doi:10.1002/jat.2949

Millstone, E. P. and Dawson, E. (2019). EFSA’s toxicological assessment of aspartame: Was it even-handedly trying to identify possible unreliable positives and unreliable negatives? Arch Public Health 77, 34. doi:10.1186/s13690-019-0355-z

Moher, D., Shamseer, L., Clarke, M. et al. (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev 4, 1. doi:10.1186/2046-4053-4-1

Morgan, R. L., Thayer, K. A., Bero, L. et al. (2016). GRADE: Assessing the quality of evidence in environmental and occupational health. Environ Int 92-93, 611-616. doi:10.1016/j.envint.2016.01.004

Morgan, R. L., Thayer, K. A., Bero, L. et al. (2018). Identifying the PECO: A framework for formulating good questions to explore the association of environmental and other exposures with health outcomes. Environ Int 121, 1027-1031. doi:10.1016/j.envint.2018.07.015

Morgan, R. L., Beverly, B., Ghersi, D. et al. (2019). GRADE guidelines for environmental and occupational health: A new series of articles in Environment International. Environ Int 128, 11-12. doi:10.1016/j.envint.2019.04.016

NRC – National Research Council (2014). Review of EPA’s Integrated Risk Information System (IRIS) Process. Washington, DC, USA: The National Academies Press.

NTP – National Toxicology Program (2015). Handbook for Preparing Report on Carcinogens Monographs.

NTP (2016). Monograph on Immunotoxicity Associated With Exposure to Perfluorooctanoic Acid (PFOA) and Perfluorooctane Sulfonate (PFOS).

OECD (2019). Guiding Principles and Key Elements for Establishing a Weight of Evidence for Chemical Assessment. OECD Series on Testing and Assessment, No. 311. OECD Publishing, Paris.

Pega, F., Momen, N.C., Ujita, Y. et al. (2021). Systematic reviews and meta-analyses for the WHO/ILO joint estimates of the work-related burden of disease and injury. Environ Int 155, 106605. doi:10.1016/j.envint.2021.106605

RAC and SCOEL (2016). ECHA Committee for Risk Assessment (RAC) and Scientific Committee on Occupational Exposure Limits (SCOEL) Joint Opinion to resolve differences in scientific opinion as regards exposure levels for N-methyl-2-pyrrolidone.

Rhomberg, L. R., Goodman, J. E., Bailey, L. A. et al. (2013). A survey of frameworks for best practices in weight-of-evidence analyses. Crit Rev Toxicol 43, 753-784. doi:10.3109/10408444.2013.832727

Robinson, C., Portier, C. J., Cavoski, A. et al. (2020). Achieving a high level of protection from pesticides in Europe: Problems with the current risk assessment procedure and solutions. Eur J Risk Regul 11, 450-480. doi:10.1017/err.2020.18

Rooney, A. A., Boyles, A. L., Wolfe, M. S. et al. (2014). Systematic review and evidence integration for literature-based environmental health science assessments. Environ Health Perspect 122, 711-718. doi:10.1289/ehp.1307972

Rooney, A. A., Cooper, G. S., Jahnke, G. D. et al. (2016). How credible are the study results? Evaluating and applying internal validity tools to literature-based assessments of environmental health hazards. Environ Int 92-93, 617-629. doi:10.1016/j.envint.2016.01.005

Rudén, C. (2001). The use and evaluation of primary data in 29 trichloroethylene carcinogen risk assessments. Regul Toxicol Pharmacol 34, 3-16. doi:10.1006/rtph.2001.1482

Samuel, G. O., Hoffmann, S., Wright, R. A. et al. (2016). Guidance on assessing the methodological and reporting quality of toxicologically relevant studies: A scoping review. Environ Int 92-93, 630-646. doi:10.1016/j.envint.2016.03.010

Schenk, L. (2010). Comparison of data used for setting occupational exposure limits. Int J Occup Environ Health 16, 249-262. doi:10.1179/107735210799160255

Schenk, L. and Johanson, G. (2011). A quantitative comparison of the safety margins in the European indicative occupational exposure limits and the derived no-effect levels for workers under REACH. Toxicol Sci 121, 408-416. doi:10.1093/toxsci/kfr056

Schenk, L., Deng, U. and Johanson, G. (2015). Derived no-effect levels (DNELs) under the European chemicals regulation REACH – An analysis of long-term inhalation worker-DNELs presented by industry. Ann Occup Hyg 59, 416-438. doi:10.1093/annhyg/meu103

Schreider, J., Barrow, C., Birchfield, N. et al. (2010). Enhancing the credibility of decisions based on scientific conclusions: Transparency is imperative. Toxicol Sci 116, 5-7. doi:10.1093/toxsci/kfq102

Schulz, K. F., Chalmers, I., Hayes, R. J. et al. (1995). Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 273, 408-412. doi:10.1001/jama.273.5.408

Schünemann, H., Hill, S., Guyatt, G. et al. (2011). The GRADE approach and Bradford Hill’s criteria for causation. J Epidemiol Community Health 65, 392-395. doi:10.1136/jech.2010.119933

Schünemann, H. J., Brozek, J., Guyat, G. et al. (2013). GRADE handbook for grading quality of evidence and strength of recommendations.

Stephens, M. L. et al. (2013). Evidence-based toxicology for the 21st century: Opportunities and challenges. ALTEX 30, 74-103. doi:10.14573/altex.2013.1.074

Stephens, M. L., Andersen, M., Becker, R. A. et al. (2016). The emergence of systematic review in toxicology. Toxicol Sci 152, 10-16. doi:10.1093/toxsci/kfw059

Thayer, K. A., Wolfe, M. S., Rooney, A. A. et al. (2014). Intersection of systematic review methodology with the NIH reproducibility initiative. Environ Health Perspect 122, A176-177. doi:10.1289/ehp.1408671

van Luijk, J., Popa, M., Swinkels, J. et al. (2019). Establishing a health-based recommended occupational exposure limit for nitrous oxide using experimental animal data – A systematic review protocol. Environ Res 178, 108711. doi:10.1016/j.envres.2019.108711

Whaley, P., Halsall, C., Agerstrand, M. et al. (2016). Implementing systematic review techniques in chemical risk assessment: Challenges, opportunities and recommendations. Environ Int 92-93, 556-564. doi:10.1016/j.envint.2015.11.002

Whaley, P., Aiassa, E., Beausoleil, C. et al. (2020a). Recommendations for the conduct of systematic reviews in toxicology and environmental health research (COSTER). Environ Int 143, 105926. doi:10.1016/j.envint.2020.105926

Whaley, P., Edwards, S. W., Kraft, A. et al. (2020b). Knowledge organization systems for systematic chemical assessments. Environ Health Perspect 128, 125001. doi:10.1289/EHP6994

Wikoff, D., Urban, J. D., Harvey, S. et al. (2018). Role of risk of bias in systematic review for chemical risk assessment: A case study in understanding the relationship between congenital heart defects and exposures to trichloroethylene. Int J Toxicol 37, 125-143. doi:10.1177/1091581818754330

Wikoff, D., Haws, L., Ring, C. et al. (2019). Application of qualitative and quantitative uncertainty assessment tools in developing ranges of plausible toxicity values for 2,3,7,8-tetrachlorodibenzo-p-dioxin. J Appl Toxicol 39, 1293-1310. doi:10.1002/jat.3814

Wikoff, D., Urban, J. D., Ring, C. et al. (2020a). Development of a range of plausible noncancer toxicity values for 2,3,7,8-tetrachlorodibenzo-p-dioxin based on effects on sperm count: Application of systematic review methods and quantitative integration of dose response using meta-regression. Toxicol Sci 179, 162-182. doi:10.1093/toxsci/kfaa171

Wikoff, D., Lewis, R. J., Erraguntla, H. et al. (2020b). Facilitation of risk assessment with evidence-based methods – A framework for use of systematic mapping and systematic reviews in determining hazard, developing toxicity values, and characterizing uncertainty. Regul Toxicol Pharmacol 118, 104790. doi:10.1016/j.yrtph.2020.104790

Wolffe, T. A. M., Whaely, P., Halsall, C. et al. (2019). Systematic evidence maps as a novel tool to support evidence-based decision-making in chemicals policy and risk management. Environ Int 130, 104871. doi:10.1016/j.envint.2019.05.065

Woodruff, T. J. and Sutton, P. (2014). The navigation guide systematic review methodology: A rigorous and transparent method for translating environmental health science into better health outcomes. Environ Health Perspect 122, 1007-1014. doi:10.1289/ehp.1307175

Yost, E. E., Euling, S. Y., Weaver, J. A. et al. (2019). Hazards of diisobutyl phthalate (DIBP) exposure: A systematic review of animal toxicology studies. Environ Int 125, 579-594. doi:10.1016/j.envint.2018.09.038

Most read articles by the same author(s)

1 2 3 > >>