Nested Knowledge

The extent to which a systematic review/meta-analysis (SR/MA) can draw conclusions about the effects of an intervention depends on whether the data and results from the included studies are valid. In particular, an SR/MA of invalid studies may produce a misleading result, yielding a narrow confidence interval around the wrong intervention effect estimate. The evaluation of the validity of the included studies is therefore an essential component of SR/MAs, and should influence the analysis, interpretation, and conclusions of the SR/MA. Standardized “risk of bias” (RoB) tools are used to measure the extent of various types of biases from individual studies.

The validity of a study may be considered to have two dimensions. The first dimension is whether the study is asking an appropriate research question. This is often described as ‘external validity’, and its assessment depends on the purpose for which the study is to be used. External validity is closely connected with the generalizability or applicability of a study’s findings. The second dimension of a study’s validity relates to whether it answers its research question ‘correctly’, that is, in a manner free from bias. This is often described as ‘internal validity’. A good meta-analysis should assess both external validity and internal validity and should incorporate measures to minimize the risk of such biases.

Below are procedures for performing RoB assessments for systematic reviews and meta-analyses.

1. It is the responsibility of the study coordinator to assign the quality control (QC) reviewer and independent RoB reviewers at the beginning of the project start. Typical RoBs require 1 QC reviewer and 2 independent RoB reviewers.

1. Different study designs (Case Control, Cohort, etc.) will use different versions of an RoB. Most RoBs will consist entirely of randomized controlled trials and/or cohort studies; however, different study designs may be included depending on the project. If necessary, discuss with the study coordinator (most likely John Pederson, jpederson@supedit.com) to determine which RoB should be used. Note that it is the responsibility of the QC reviewer to determine the correct RoB form.
2. Familiarize yourself with the different sections of the RoB you are using. Information on the different RoBs can be found in the Risk of Bias folder of the SME Manuscripts Team Drive. The common RoBs are,
   1. Scottish Intercollegiate Guidelines Network (SIGN) - Systematic Reviews and Meta-analyses
      1. Randomized Controlled Trial
      2. Cohort
      3. Case Control
      4. Diagnostic Accuracy
      5. Economic Evaluations
      6. NOTE: The instructions for the SIGN RoB are quite long (100+ pages!). The good news is that most of the information is not particularly relevant for the majority of SR/MAs. For quick reference, the most important sections to review are page #51 and pages #56-66. Other sections may be useful depending on the project.
   2. modified Newcastle-Ottawa Scale (mNOS)
      1. Cohort
      2. Case Control
   3. Joanna Briggs Institute (JBI)
      1. Case Report
      2. Case Series
   4. Cochrane
      1. Randomized Trial
      2. Non-Randomized Study
3. Know which study designs you are including in the SR/MA. Be familiar with these study designs and choose the correct RoB template based on the studies included in the SR/MA.
4. Normally, two RoB reviewers will complete an RoB individually. Do not collaborate on one with your fellow reviewer because your RoB assessment would be biased.
5. An experienced QC reviewer should adjudicate results of the RoB and resolve any conflicts between RoB results from the independent reviewers.

1. Use the correct template for each study. Different study designs warrant the use of different templates. All templates exist as both Excel files and Google Sheets in the Risk of Bias folder of the SME Manuscripts Team Drive. (If any updates to the Excel file RoB template are made, please update the corresponding Google Sheet RoB template as well, since only the Google Sheet templates live in the Google Sheets Template Gallery, which are used to quickly and easily create new RoB’s in project folders by clicking New > Google Sheets > From a template.) It is the responsibility of the QC reviewer to determine which template to use.
2. SIGN Template: Use the SIGN template by default (see below sections for exceptions). SIGN includes templates for various study designs, including:
3. Systematic Reviews and Meta-analyses
   1. mNOS Template: This template includes a page for Cohort and for Case Control studies. Use mNOS template by default when the SR/MA is ONLY comprised of case control and/or cohort studies OR if requested by a client OR if requested by a journal/reviewer.
   2. JBI Template: This template includes a page for Case Reports, Case Series, Cohort Studies, Randomized Controlled Trials, and Quasi-experimental studies. Use the JBI template by default when the SR/MA involves any case reports and/or case series OR if requested by a client OR if requested by a journal/reviewer.
   3. Cochrane Templates: The Cochrane Templates have the most difficult instructions and will most likely not be used. The Cochrane Templates are applicable for both randomized studies (see RoB 2) and non-randomized studies (see ROBINS-I). Use the Cochrane template by default ONLY if requested by a client OR if requested by a journal/reviewer.

Before completing the RoB assessment, always ensure that you have the same study design labels and study row ordering as the other independent reviewer. Once this has been confirmed, perform the RoB assessment. Email the QC reviewer and study coordinator when finished, then start on data collection of each included study.

• Note: If independent reviewers create their RoBs using the New > Google Sheets > From a template route, this helps ensure that independent reviewers have uniform formatting in their RoB forms, which eases the process of response comparison for the QC reviewer during adjudication.

1. First, combine and compare indepedent RoB responses. This can be performed manually for small projects, but for larger projects, it is recommended to use the methods described in RoB Match Determination in Excel.
2. Adjudicate.
3. Write up the RoB results in the manuscript. See RoB Write-Up: Examples and Guide for how to present RoB information in the manuscript. Email the study coordinator when this process is complete.