Systematic Reviews are most frequently conducted in the fields of education, health & medicine, social policy or psychology; even if you aren't conducting a Systematic Review, they can be useful as the authors will usually published a detailed search strategy alongside the main findings, which can be helpful in seeing how other academic authors have constructed their search strategy. Durham University Library does not currently offer a dedicated systematic review support service. However, the resources collected here offer practical guidance and support on:
"A Systematic Review is a literature review that is designed to locate, appraise and synthesise the best available evidence relating to a specific research question and provide informative and evidence-based answers." (Boland et al, 2014)
It aims to review the available research evidence with the same level of rigour that should be used in producing the research evidence in the first place (Hemingway & Brereton, 2009), taking steps to "reduce/make transparent hidden bias and 'error'" (Newman & Dickson, 2012) and allowing "potentially unmanageable amounts of literature to be managed in a scientifically credible and reliable way (Torgerson, Hall & Light, 2012).
To achieve this, a Systematic Review seeks to:
The end goal is to produce more reliable findings that can be used to inform future decision-making in that field (Higgins & Green, 2011).
A Systematic Review is not just a 'big' literature review. Whilst both seek to provide a summary of the available literature on a topic, a Systematic Review is expected to be more rigorous, and ultimately to be transparent and replicable.
| Systematic Review | Literature Review | |
|---|---|---|
| Objecties | Focussed on a single, pre-determined research question, with clear objectives identified and stated prior to conducting the primary search of the literature. | Not necessarily focussed on a single research question. May describe an overview around a theme or topic of research. Clear objectives for the literature search may not be identified in advanced, but may be shaped during the review of the literature itself. |
| Preparation | Methods of identification, selection, evaluation and synthesis are pre-defined to minimise the risk of bias. | Methods of identification, selection, evaluation and synthesis may change subject to authors changing awareness and knowledge of evidence presented, building in the authors subjective knowledge and bias into the review. |
| Design & Methodology | Based on clear identifiable steps with the purpose of being replicable. | No clear rationale; based on expert substantive knowledge of author(s). |
| Selection of Evidence | Explicit over how reviews included were identified and selected. Intended to search exhaustively, and account for, all relevant studied. | Selection is not made explicit. It may be unclear whether selected evidence is representative, comprehensive or a (biased) sample. |
| Evaluation of Evidence | All assumptions and judgments are made explicit against identified criteria and open to scrutiny and replication. | Subject to author(s) knowledge and opinion. |
| Outcomes & data synthesis | Clear summary of studies, including an objective assessment of the quality of the evidence presented and potential for bias in the studies selected. | Summary based on studies may not address in detail quality of data or presence of potential bias. |
The above table was based upon comparisons found in Torgerson, Hall and Light (2012) (in Arthur et al. (2012)) and Bettany-Saltikov (2012).
Any Systematic Review should be approached in a methodical way, with several key steps outlined below. There are various ways this process is outlined in the literature, although all roughly following a similar process.
Below we have tried to provide some useful examples of systematic reviews where you can access and review the search strategy employed. This may be of interest when approaching your own Systematic Review, but also even if you are not engaged in a systematic review, as visible examples of how other researchers have employed various search tools and techniques to construct a well-focussed but comprehensive search strategy.
When approaching Systematic Reviews, it is important to be aware of the limitations of using Google Scholar for this purpose - and clearly identify if and how you have used it as a tool to support your search strategy.
If a systematic review aims to locate all of the best available evidence relating to a specific research question, then a key aim of the researcher should be to search multiple appropriate sources to ensure a comprehensive search of the available literature.
Google Scholar might seem a logical tool to use, given its breadth of coverage. However, another criteria of a systematic review is transparency in how available research is identified and selected. Here Google Scholar creates problems for the researcher in the context of a Systematic Review:
One advantage Google Scholar does offer is its coverage of grey literature, which may not be as well indexed or covered by traditional academic databases. So Google Scholar is often used to plug these gaps in a search protocol. A useful tool for making the most of this is Publish or Perish, which more easily allows you to collect, re-order, review, filter and export your results identified from a Google Scholar search.
Google Scholar cannot accurately process any search which uses more than one “concept”, each described by multiple synonyms/alternative terms.
For example:
... will be interpreted by Google Scholar as...
This can have a significant effect on the ability to identify and filter the most appropriate results. This deficiency is exaggerated further where you cannot view any results past the first 1,001 results identified, when they are ranked by Google Scholar's proprietary (and unknown) search and ranking algorithm.
Some studies (Gehanno et al, 2013) highlight that publications found through searching other academic and professional sources can also be found via Google Scholar, and have argued that Google Scholar is therefore a useful source for Systematic Reviews as it's coverage is much broader than other tools. However, other researchers have pointed out that searching for known articles is very different to constructing a comprehensive search protocol using boolean operators to identify unknown research studies.
Search results will be ranked by Google Scholar's proprietary (and unknown) search and ranking algorithm. Whilst unknown, it is apparent that the ranking takes into account various factors, including:
... but also
Given this, it is possible that a search protocol will offer different results, and ranking of results, to different users. This, alongside the problems with the search logic and the lack of a clear (and changing) coverage list, make it difficult to reproduce a search on Google Scholar with the level of certainty a Systematic Review is expected to provide.
One key problem you may face is that the same study may be found in more than one database searched. A single study published in the journal Nature, for example, might be indexed in Web of Science, Scopus, Medline, Embase and PsycINFO. You will need to de-duplicate your list of studies so that duplicate studies are removed from your list of results, both to save time and to correctly record in your protocol or PRISMA flow diagram the actual number of unique studies identified.
