OHSU

Methods

DERP SYSTEMATIC REVIEWS

A systematic review uses rigorous, unbiased methods to examine the similarities and differences of results across many individual studies. A systematic approach to reviewing the evidence increases the reliability of the results, and the transparency of the procedures. The goal of a DERP systematic review is to estimate the comparative effectiveness and safety of healthcare treatments.Nomination, selection and definition of drug classes are directed by the DERP Participants using a self-governing process.  

  • Disease burden
  • Alternatives
  • Scope
  • Clinical impact
  • Budget impact
  • Evidence
  • Marketing
  • Benefits/Policy Considerations
The latest version of the Review Methods and Report Production Procedures for the Drug Effectiveness Review Project is available here

The method for producing a systematic review is composed of: 

Key Question Development

Literature Search 

Articles Selection 

Quality Standards 

Data Inclusion, Synthesis and Reporting 

Peer Review and Public Comment 

Updates  


KEY QUESTIONS

Formulation of key questions and inclusion criteria is the first step of a systematic review and specifically defines the scope of the investigation. The key questions and inclusion criteria provides specific instructions to the reviewer on each aspect of the review scope: populations, interventions, outcomes and study designs. In this project, the key questions are written by the Evidence-based Practice Center (EPC) in consultation with the Participating Organizations, who work with content experts to ensure that the questions are clinically relevant. Prior to becoming a final draft, key questions are posted for public comment and revised as necessary. After key questions have been finalized, they are posted in the Final Products section of the site for easy reference.

LITERATURE SEARCH

An important factor that distinguishes a systematic review from a traditional review is the way that studies are located. In a traditional review, an author will usually search a large medical database, such as Medline, and will include well-known studies and studies by researchers he or she is familiar with. This method may miss important studies, because not all medical journals are included in Medline, and even the most knowledgeable researcher will not know about every study that has been conducted on a topic. In a systematic review, a more thorough search is conducted. After developing a search strategy with a medical librarian who has special expertise with systematic reviews and the medical literature databases, multiple databases are searched. Authors also hand search reference lists of published studies, and solicit additional citations from content experts and pharmaceutical companies. All of the search strategies are published with the reports, so the methods used to locate studies are apparent and may be reproduced.

ARTICLE INCLUSION CRITERIA

Deciding to select an article for inclusion in the review is a multiple-step process and is ultimately made based on the prespecified inclusion criteria provided in the key questions. First, potentially relevant articles are identified for retrieval based on a preliminary assessment of the patient population, treatment, efficacy and safety outcomes, and design characteristics provided in titles and abstracts (if available) of citations. Retrieved articles are then again assessed in their entirety for these characteristics, as well as rated for their methodological quality.
Only articles that report about studies that were rated as being of "fair" or "good" quality are included. (Please see the "Quality Assessment" section for details about rating methods.) Generally, at least two independent reviewers will be involved in the selection process. All disagreements between the reviewers are resolved by consensus. The rigor of this process and the use of dual review are both factors that reduce the chance of introducing bias.

QUALITY

The quality (or internal validity) of all included studies is assessed independently by two reviewers, with disagreements between them resolved by consensus. Internal validity is the degree to which the study (as designed, conducted and analyzed) represents the "true" results. The methods used to enroll patients, assign patients to treatment groups, assess outcomes, and analyze data can all affect the results of a study. When the design of a study leads to errors that distort the true effects of a treatment, that study is considered biased; bias can be intentional or unintentional. Research has shown that certain factors, such as whether investigators know which treatment a patient is assigned to, are likely to lead to biased results. Careful attention is paid to these factors to determine how they might affect results.
Studies that are biased are not used by DERP authors to draw conclusions. All studies are assigned a rating of Good, Fair, or Poor according to prespecified criteria. The results of a study rated Good are likely to be valid, and the results of a study rated Poor are not valid. The Fair category is broad, with some studies in this group more likely to be valid than others. The details of quality ratings of all included studies are provided in the reports. External validity (or generalizability) refers to the ability of an individual study to apply to a given patient population in a natural setting. Because the generalizability of a study varies with the population the study results are being extrapolated to, external validity is not rated as good, fair, or poor.

For specific criteria used in quality assessment, see this quality assessment PDF.

DATA

Data (such as study design, patient characteristics, interventions, methods of outcome measurement, and results) from individual studies within a review are collected and recorded in a standardized way. Abstraction is done independently by at least two reviewers. Abstraction using two reviewers reduces bias and improves data accuracy. This information is then displayed in evidence tables in the final report to allow the reader to make direct comparisons and assessments across categories.

An example of an evidence table (PDF) can be found here.

DATA SYNTHESIS AND REPORTING

Data reporting involves generating a summary of study characteristics, methodological quality, and results. Once all of the included studies have been assessed for quality, and data has been abstracted, the data are synthesized to address the key questions.

Evidence tables provide the structure for examining the similarities and differences among these variables across the studies. The Evidence Tables facilitate synthesis of quality, characteristics, and the results of each study. A narrative synthesis takes many factors into account to provide the reader with an overall understanding of the evidence relating to each key question. Formal statistical techniques for pooling results from multiple studies (meta-analysis) are also used, when appropriate, to calculate pooled estimates of effect. 

More information about meta-analysis can be found in: Lau J. Ioa Ioannidis JP, Schmid CH "Quantitative Synthesis in Systematic Reviews," Annals of Internal Medicine. 127(9):820-6, 1997. 

DERP reports include an extensive narrative text that details the review rationale and objectives, methods, and results, and are accompanied by reference lists, evidence tables, and various supporting figures and appendices.

PEER REVIEW & PUBLIC COMMENT

Before becoming final, reports undergo a peer review process. The draft manuscripts are sent to experts in the clinical area being reviewed requesting their evaluation of and comment on our review. Peer reviewers can comment on any aspect of the systematic review. Comments from peer reviewers are considered in the final report. To participate in the peer review process registration is required. Click here to see a list of Peer Reviewers.

UPDATES

Reports are considered for update on an annual basis. The consideration process includes a scan of the literature using the same search strategy as the Final Report but limited to Medline. In addition, new drugs, safety alerts, and indications in the drug class are identified. This (along with all abstracts of articles identified in the search) are reviewed by Participating Organizations, and a decision is made whether or not to proceed with an update. If a decision is made to update a report, the key questions are reviewed for relevance and for potential changes. The process then repeats in the same fashion as an original report.