Below you can find my current (version 0.6) recommended practices for how to conduct a systematic review/map in SE. The basic starting point for a sysrev in SE is the Kitchenham report.
- Hybridize the normal (Kitchenham) sysrev with snowball sampling. Use a sysrev search, screening and paper selection for the last, full 10 years. Then study the reference lists of the final list of selected papers and be sure to include all the relevant papers also in earlier years. This way the systematic search for the last 10 years of results are used as a "lens" to ensure the best studies of the previous years are also included. Reference the original snowball sampling paper by Leo Goodman from 1961. Of course if the investigated area is small, don't restrict the initial search to 10 years.
- Select databases wisely. Google Scholar gets better by the day even if it is still "less formal" than the main ones. Scopus has grown to be one of the largest. IEEE Xplore and ACM Digital Library are almost necessary to include in SE, but ACM is harder to use in practice. Having less than 4 databases is uncommon, as is having more than 6. IMHO, Compendex, ISI/Web of Science, and Scopus tend to give very similar results so not a must to include all.
- Explore search strings systematically until you get down to a manageable number of hits (typically 500-8000 initial hits which will typically result in 50-400 finally included papers which is typically manageable (for more specific SLR's these numbers can be 10-50% lower)).
- Yes, you should not really adapt the search strings based on the number of hits but in reality you cannot perform a sensible or publishable sysrev with too few or too many hits.
- Another reasonable way to reduce the number of hits if the investigated area is large is to only include journal papers.
- Systematic way to devise and validate searches with the quasi gold standard: Zhang et al 2011.
- To validate the search string it is recommended to have a validation set of
relevant papers (typically 10-20), which would be included in the review.
This set should be as heterogeneous as possible (different authors, publication
venues, research groups/universities) to avoid bias.
The search string can then be exercised against this known set in order to
test if it captures all publications.
The search string should *not* be refined by explicitly using the keywords from
papers in the validation set since that would again bias the results. Try to
think and identify new keywords on your own (ask experts on the topic under
investigation, use SWEBOK [1] to find specific terminology).
- Use Inter-rater Agreement Analysis for both the screening and data extraction phases. To ensure objective selection of papers as well as the right data extraction and paper classification. Use Cohen's Kappa, Krippendorf's Alpha or Gwet's AC1/AC2 statistics for this (the latter are the best and very general, Krippendorff is as general but has been around longer and Cohen's Kappa is still the most common).
- Include standard attributes in the form used to extract data about each paper. Examples are:
- Map the results using bubble graphs as in the original sysmap paper by Petersen, Feldt et al. Another early use was in Afzal et al's sysmap.
- Do extensive result summary and synthesis in the style of the excellent review by Yoo and Harman.
- Contact the main authors in the field (the ones with the most papers among your finally selected ones) via email. Ask them
- if they have published or know of other related work in the area, and
- if they want to comment on the part of your text where you later summarize their papers.
Before you email anyone though you have to clear your text and what you write in the email with your supervisor! Some researchers might get many requests like this and it has to be balanced in the right way.
- When designing your sysrev and before submitting it, ensure you cover the PRISMA checklist for reporting on systematic reviews. It is from medicine and focused on experiments but still is valid for other types of studies. More info on PRISMA here.
Thanks to my students and colleagues that helped refine this over several sysrev studies. Please contact me with any comments or corrections you might have to this list.