Skip to main content
The HS/HSL is a part of the University of Maryland, Baltimore | My UMB The Elm UM Shuttle Google Apps@UMaryland Blackboard

601 West Lombard Street
Baltimore MD 21201-1512

Reference: 410-706-7996
Circulation: 410-706-7928

Systematic Review Service

The Process

The following information is intended to be a general introduction to the process of conducting systematic reviews.  For further details and guidance, refer to the Standards section of this guide.

For a video overview of the systematic review process, visit our tutorial.

Determine if a Systematic Review is Necessary

Before starting your review, determine if a systematic review is the best approach to answer your question. 

  • Will it fill a meaningful gap in existing literature?
  • Is there already an existing review? 
    • If you identify an existing review, assess its quality. (see box below)
    • If a high quality review exists but was completed several years ago, a new review may be justified.
  • Do you have the necessary time and resources to complete a systematic review?

 

From: Centre for Reviews and Dissemination, University of York (2008) Systematic Reviews: CRD's Guidance for Undertaking Reviews in Health Care.

Clarify the Question

A clearly defined, focused review begins with a well-framed question.  It provides the framework for the entire review.

  • Clearly state the objectives of the review (what question are you trying to answer?)
  • Consider the PICOS framework
    • Population
    • Interventions
    • Comparisons
    • Outcomes
    • Study Design

Systematic Reviews are not limited to questions about effects of interventions, they may address trends, accuracy of diagnostic tests, effectiveness of programs, etc.

Ask yourself...

  • Is the question answerable?
  • Are there measurable constructs?
  • Is it practical or relevant for policy/practice?

 

Define Exclusion/Inclusion Criteria

One of the features that distinguishes a systematic review from a narrative review is the pre-specification of criteria for including and excluding studies in the review (eligibility criteria). Explicit criteria, based on the review’s scope and question(s), are used to include and exclude studies. 

A large number of references (study titles and abstracts) will have been found at the searching stage of the review. A proportion of these will look as though they are relevant to the review's research questions. So, having explicit criteria against which to assess studies makes the process more efficient in terms of time.

More importantly, it also helps to avoid hidden bias by having clear consistent rules about which studies are being used to answer the review's specific research questions. 

Each study needs to be compared against same criteria. To be included in the review, a study needs to meet all inclusion criteria and not meet any exclusion criteria. Inclusion/eligibility criteria iclude participants, interventions and comparisons and often study design. Outcomes are usually not part of the criteria, though some reviews do legitimately restrict eligibility to specific outcomes.

Create a Protocol

The review protocol sets out the methods to be used in the review and provides an explicit plan for your work. Decisions about the review question, inclusion criteria, search strategy, study selection, data extraction, quality assessment, data synthesis and plans for dissemination should be addressed. If modifications to the protocol are required, these should be clearly documented and justified. Modifications may arise from a clearer understanding of the review question, and should not be made because of an awareness of the results of individual studies. 

Consider registering your protocol. PROSPERO, from the Centre for Reviews and Dissemination at the University of York, is an international database of registered reviews in health and social care.  Key features from registered reviews and recorded and maintained as a permanent record.  Registration helps to promote transparency in the review process and also reduces the potential for duplication.

Search for Studies

A systematic search to identify studies must be comprehensive and it must strike a balance between recall and precision. In other words, don't expect to retrieve only relevant articles. "Being comprehensive means that the search strategy attempts to uncover published and unpublished, easily accessible and harder to find reports of research studies. Bias can creep in if the search is not exhaustive in this way as, for example, statistically significant positive results are more likely to be published and cited by others." (EPPI-Centre Methods for Conducting Systematic Reviews)

Most studies used in the review will be identified using electronic databases (e.g. PubMed). But, The Cochrane Collaboration states that "efforts should be made to identify unpublished studies." (Cochrane Handbook 6.2.4)  Librarians can help to identify sources for finding grey literature.

Expect a large number of results.  Depending on the scope of your topic, your search may result in as few as a couple hundred or as many as several thousand articles.

Some key steps in searching for studies include:

  • Identifying databases to be searched.
  • Identifying search terminology.
  • Constructing and running database searches.
  • Conducting hand searches of specialized journals.
  • Searching reference lists of relevant studies.
  • Contacting recognized experts working in the field.
  • Searching relevant grey literature sources (e.g. clinical trials registers, conference proceedings).

It’s important to note that the IOM recommends working “with a librarian or other information specialist trained in performing systematic reviews to plan the search strategy” (IOM Standard 3.1.1).

Librarians at the HS/HSL are expert searchers who can support faculty investigators in conducting comprehensive literature searches for systematic reviews, assist with reference management and writing the search methodology section of the review.  Consider meeting with a librarian to discuss your systematic review project.

Select Studies

A large quantity of studies needs to be assessed at this stage of the review.  This process should involve at least two members of your group to help reduce bias.  It is useful to develop an appropriate form to help select and keep track of articles that meet eligibility criteria. To be included, a study must meet all eligibility/inclusion criteria and not meet any exclusion criteria. 

Employ the following process:

  1. Examine titles/abstracts
  2. Retrieve the full-text of relevant studies
  3. Examine the full-text to determine eligibility
  4. Make final decisions on study inclusion

Remember to document your decisions!  It is important to have a record of decisions made in order to ensure reproducibility and minimize errors.

This PRISMA flow diagram shows the number of studies/papers remaining at each stage. It's a simple and useful way of documenting the study selection process and should be included in your final report.

Extract Data

Here, data simply refers to information about or details from a study including its methods and design, participants, setting, interventions, results, etc.  Ideally, two researchers should independently extract data from each study in order to minimize errors and reduce the potential for bias.  As a minimum, one researcher should extract data with a second person checking for accuracy.  

To standardize the process and improve the validity of the results, time and thought should be gievn to creating a data extraction form. These can be created using general word processing software or spreadsheets. There are also some software tools designed specifically for the data extraction process (see the Tools section on this guide), though these are often cost-prohibitive.

It is important to collect enough information, during this process, to sufficiently assess each study.  Collecting too much or too little information may be a waste of time and result in the omission of crucial data.

Assess the Quality of Studies

Assessment of study quality gives an indication of the strength of the evidence provided by your review. It helps determine whether the studies are vigorous enough to guide treatment, prevention, diagnostic or policy decisions.

Quality assessment of any study is likely to consider:

  • Appropriateness of study design to the research objective
  • Risk of bias
  • Other issues related to study quality
    • Choice of outcome measure
    • Statistical issues
    • Quality of reporting
    • Quality of the intervention
    • Generalizability

From: Centre for Reviews and Dissemination, University of York (2008) Systematic Reviews: CRD's Guidance for Undertaking Reviews in Health Care.

Synthesize Data and Write the Report

Synthesis involves combining the results of the studies included in the review, summarizing their findings and drawing reliable conclusions based on the quality of the evidence. Synthesis may be done quantitatively using statistical techniques, such as a meta-analysis, or through a narrative approach.  

In general, making recommendations for practice does not fall within the purview of systematic reviews. This is typically the domain of clinical practice guidelines. Systematic review authors can make conclusions about the need for further research or draw conclusions about the usefulness of an intervention.

When preparing your report or article, refer to the PRISMA Checklist.  Many commissioning bodies and journals have adopted PRISMA as the required methods for reporting systematic reviews.