Systematic Reviews

Selecting Studies

The Cochrane Handbook recommends that a typical process for selecting studies for inclusion in a review should be as follows:

1. Collate all your search results using reference management software (such as RefWorks or Endnote) and remove any duplicate records.

2. Scan the titles and abstracts of your references, removing any that are obviously irrelevant.  Articles and studies should be included or excluded based on the criteria set out in your protocol.

You should consider the following types of question:

• Does the study cover the correct population?
• Does it include the intervention outlined in the review question or protocol?
• Is the outcome the same as the one you are examining in your review?

Ideally, the papers should be screened by at least two people, with the option to refer to a third where there is no agreement.  Students conducting their own review will have to rely exclusively on their own judgment.  If you have any doubts about whether to include a study, do not discard it at this stage.

3. Retrieve the full text of the remaining potentially relevant papers. 

4. Check for, and then link together, any different papers which are multiple reports of the same study.

5. Check the full text papers to see if the study complies with your eligibility criteria.  Keep notes of the studies you have excluded and the reasons why, as you will need to record this information in your PRISMA flow diagram (see our Searching systematically page).  Ideally, at least two reviewers should evaluate the methodological quality of the full text papers.  Use a checklist (such as CASP) to determine whether the studies meet the criteria outlined in your protocol.  You should consider the following types of question:

• Were patients assigned to groups using random allocation;
• Was a double-blind allocation sequence used, i.e. neither patients nor clinicians were aware of their allocation;
• Is it possible that the study conclusions could have been influenced by bias?

6. Contact the authors, where necessary, to request more information on their methods or results in order to clarify whether their study meets your eligibility criteria.  Record any studies that are unobtainable, or where you cannot obtain all the necessary details, as incomplete.  List these in a table of ‘Studies awaiting assessment’ in your review.

7. Finalise which studies will be included in your review.  Begin the data collection.  It is preferable to have at least two reviewers extracting data.  Use a recognised data extraction template, such as CASP, which you can adapt for your own review.

8. Add details of any ongoing trials which have not yet been reported, into an ongoing studies table.

Potential Screening Tools

Excel

Excel is the most basic tool for the management of the screening and data extraction stages of the systematic review process. Customised workbooks and spreadsheets can be designed for the review process. A more advanced approach to using Excel for this purpose is the PIECES approach, designed by a librarian at Texas A&M. The PIECES workbook is downloadable here.

Rayyan

Rayyan is a free web based screening tool for systematic reviews. There is a video tour of Rayyan available here: https://www.youtube.com/embed/irAOQgzFMs4

Reference Management Software 

Reference Management software allows you to organise, store and manage your references. At Teesside, EndNote and RefWorks are available to use:

• EndNote
• RefWorks

Critical Appraisal

Once you have selected the relevant studies you must evaluate their quality as not all will have a sufficiently rigorous methodology to avoid biased results. 

Checklists

Use the checklists available via the links below to help you with this task. You may need to explore a few before you find one which is best for your needs.

AMSTAR (A MeaSurement Tool to Assess systematic Reviews): AMSTAR is an instrument used in assessing the methodological quality of systematic reviews which has produced a checklist to help with this.

Centre for Evidence-Based Medicine (CEBM): This Critical Appraisal Tools webpage contains useful checklists/worksheet to help with the critical appraisal of different types of medical evidence.

Critical Appraisal Skills Programme (CASP): This set of eight critical appraisal tools are designed to be used when reading research, these include tools for Systematic Reviews, Randomised Controlled Trials, Cohort Studies, Case Control Studies, Economic Evaluations, Diagnostic Studies, Qualitative studies and Clinical Prediction Rule. The checklists can be printed or filled in electronically. The University of Sheffield have produced a video to explain how to use the CASP checklist.

Duke University Medical Centre: The Evidence-Based Practice: Appraise section of this guide includes a variety of critical appraisal worksheets/checklists. 

Joanna Briggs Institute (JBI): This Critical Appraisal Tools webpage from the JBI provides a variety of checklists for different types of research/study design.

Mixed Methods Appraisal Tool (MMAT): The MMAT critical appraisal tool is designed for the appraisal stage of reviews that include qualitative, quantitative and mixed methods studies. It facilitates appraisal of the methodological quality of qualitative research, randomized controlled trials, non-randomized studies, quantitative descriptive studies, and mixed methods studies.

NICE Quality Appraisal Checklist - Qualitative Studies: This qualitative checklist is designed for people with a basic understanding of qualitative research methodology.  It can be used for studies which collect and analyse qualitative data, usually (but not exclusively) textual (written), spoken or observational data.

PEDRO Scale: The PEDro scale is designed to help people using the PEDro database identify which of the known or suspected randomised clinical trials archived on the PEDro database are likely to be internally valid (criteria 2-9), and could have sufficient statistical information to make their results interpretable (criteria 10-11). 

PRISMA: The PRISMA checklist is a 27-part list of the qualities a high-quality paper should contain. Using it you would appraise the quality of a paper's structure, methodology, reporting and more. You can use it on your own paper to check where you need to improve.

Scottish Intercollegiate Guidelines Network (SIGN): SIGN produces a number of checklists.  These give a framework of questions to apply to each study to judge the likelihood of bias.

Critical Appraisal Worked Examples

This useful video from the Cochrane Common Mental Disorders on critical appraisal, is the first in the series of seven videos. It discusses the key concepts of critical appraisal. The full series can be found here. In each video, they walk through the CASP checklist alongside an open access research article, to demonstrate how to approach critical appraisal in practice.

This video, created by ScHARR, University of Sheffield MPH by Distance Learning, shows a demonstration of using a CASP checklist on Randomised Controlled Trials.

This video goes through the critical appraisal steps using an example qualitative article and the CASP checklist. It has also been created by ScHARR, University of Sheffield, MPH by Distance Learning.

This PowerPoint from UCL, gives a worked example of critical appraisal of a paper: Session 5: how to critically appraise a paper – an example. 

Critical Appraisal Additional Tools and Resources

There are lots of additional online resources which explain the process of critical appraisal for a variety of resources.

AGREE Enterprise Tools:  The Appraisal of Guidelines for Research and Evaluation (AGREE) Instrument evaluates the process of practice guideline development and the quality of reporting. A variety of tools have been developed to assist in the development, reporting and evaluation of practice guidelines and health system guidance.

Claire Beecroft has produced a Prezi on critical appraisal in systematic and narrative reviews which covers: using PICO; checklists; and qualitative studies.

Colorado Community Colleges Online: Useful site for evaluating sources. The resources look at how to evaluate sources, including details about the CRAP (Currency, Reliability, Authority, Purpose/Point of View) test

Critical Appraisal for Health Students: LibGuide from Teesside University. This guide is aimed at health students who are required to undertake critical appraisal of research. It provides basic level support for appraising qualitative and quantitative research papers. It's designed for students who have already attended lectures on critical appraisal. Frameworks to appraise both types of research are provided along with an opportunity to practise appraisal skills using a research paper with suggested answers.

How to read a paper: The chapters in Trisha Greenhalgh's book, How to read a paper, explain how to read and interpret different kinds of research papers.

Idaho State University Library video on appraising the literature and a list of critical appraisal tools, linked to the websites where they were developed. 

King's College, London have created two videos on critical appraisal: 

• Critical Appraisal 2020 - Quantitative Introduction (includes: definitions of critical appraisal and quantitative research; the importance of critical appraisal to evidence-based medicine; awareness of potential biases found in quantitative research).
• Critical Appraisal 2020 - Qualitative Introduction (includes: definitions of critical appraisal and qualitative research; the importance of critical appraisal to evidence-based medicine; the application of qualitative research to real world situations.)

OT Seeker Tutorial: Tutorial on critical appraisal which includes information on: an introduction to critical appraisal; randomised controlled trials; and systematic reviews.

Specialist Unit for Review Evidence: The resources from the Specialist Unit for Review Evidence at Cardiff University include useful definitions, manuals, checklists and guides for systematic reviewers. 

UCL Great Ormond Street Institute of Child Health: This document on the critical appraisal of a journal article provides a useful summary explaining the process of critical appraisal. Gives a glossary of relevant statistical terms, links and a bibliography.

Understanding Health Research Tool: Tool designed to help people understand and review published health research to decide how dependable and relevant a piece of research is. It gives pointers towards the sort of questions to ask about a paper.

University of Suffolk: The advanced literature search guide for nursing and health sciences has a section dedicated to critical appraisal. There a links to resources and checklists as well as an introductory 10 question quiz to see how much you already know about critical analysis. 

Critical Appraisal Tutorials

The E learning modules by Amanda Burls and Anne Brice, on Finding and Appraising the Evidence, take you through the process of how to find the evidence and then how to assess the validity and reliability of the published research in order to provide effective and efficient healthcare. In each case there is a tutorial and a paper that you can download to appraise and then check your answers against the ones on the site.

• Overall introduction to critical appraisal
• Finding the evidence
• Randomised controlled trials
• Systematic reviews
• Economic evaluations
• Making sense of results

The University of Nottingham have also created a tutorial on Critical Appraisal of Clinical Trials in Dermatology, which includes a worked example of appraising a paper. You are asked to read an article on a dermatology clinical trial and then work through some interactive activities.