Dissertations & projects: The process of reviewing

When reviewing the literature, it is important to make sure you do not produce a "she said, he said" list of what the authors of the papers did and found. Reviewing involves searching, selecting, evaluating and synthesising the literature not just reporting on what is says.

Searching the literature

Whether you are undertaking a traditional or systematic literature review it is important that you approach your searching in an organised and methodical manner.

• Search appropriate academic journal databases.* Suitable databases for your subject can be found in the Databases: by subject page on the library website.
• Develop a search strategy that will look for the key concepts in your question. This includes deciding and refining your search terms but also initial filtering such as by language or date published. Help with Planning and saving a search strategy is found in the next section of this guide. 

Selecting the papers to use

*For postgraduate students and academic staff undertaking a systematic review, you will also need to search grey literature (see the Systematic Reviews Guide). This is not necessary for undergraduate reviews, even systematic ones. 

Evaluating and synthesising the literature

Although these are given here as two different processes, they are often done together. There is no right or wrong order to do this in either. Some times you will evaluate several sources and then bring them together through synthesis, other times you will synthesise several sources together and then evaluate them as a group.

Evaluating the literature

There are two main ways of evaluating literature:

1. Evaluating the worth of an individual paper
2. Evaluating how useful a paper is to your project or argument

Both are important but many students do too much of the first and not enough of the second.

Evaluating the worth of an individual paper

Not all published or even peer reviewed papers are perfect. Evaluating a paper means looking for both strengths and possible weaknesses or limitations. Sometimes the author has helped you by pointing out what they see as the limitations of their own work (usually after the discussion section) but often you have to decide on these yourself. Approach this with confidence, your thoughts matter. It is part of the process of academic research and authors expect it to happen.

When evaluating a paper, ask yourself the following questions:

Who is the author and what are their credentials/affiliations? Here you are looking to see if the author is credible and making sure they do not have any biases because of their personal interests. It is worth quickly googling them to see if they are still at the same institution (or have moved into industry or something where their findings may mean they gain financially for example). It is also worth googling their institution. Not all have the best reputations or are without their own agendas.

How reputable is the journal and how well received is the article? Check out the journal's impact factor/ranking (see either SJR or Scopus) to see if it is prestigious or influential. Check it out in relation to other journals in your subject area rather than overall (some disciplines such as medicine have much higher impact factors). Also check how many times the article has been cited (you can do this in Web of Science or get a rough idea in Google Scholar). Don't use this as a definitive guide however - some poor papers are cited a lot to point out how bad they are and older papers have obviously had more time to be cited.

Are the results reliable? Are the conclusions backed up by the evidence? Was the methodology suitable? Were there any design or analysis flaws? In quantitative studies, is the sample size big enough for generalisation? Was there a control group? Were participants assigned to groups appropriately? In qualitative studies, were interviews piloted? Does the researcher declare their relationship with the participants? Were results triangulated for verification?

Evaluating a paper's usefulness to your argument or study

One of the most important parts of a literature review is deciding which papers are the most relevant to your overall argument (for stand alone reviews) and/or to position your study within the rest of the literature (for research papers, independent projects, dissertations or theses). Remember for a literature review your argument is about the literature rather than supported by it as in an essay.

Here are some questions you could ask yourself: 

Is the article current (or timeless)? Decide if the article is up-do-date and check to see if has it been superseded by more recent research. In Google Scholar or Web of Science for example, you can see which other articles have cited a particular paper so you may be able to quickly see if there is more recent research on the topic. In some disciplines, older, seminal works may be considered timeless and still relevant. If you are structuring your review chronologically you will be specifically looking for papers from particular time frames.

How relevant is it? This is very subjective and can depend on how much literature there is on your topic. If there is a lot, you can afford to be very specific about the context you are looking for. For example, in business, is it concerned with the right sector? In education, is it concerned with the appropriate level of study? If there is very little specific literature, you will need broader criteria. For example, is it in a related field? Is it using a similar methodology?

Does it/could it support the argument I am developing? As you scope out the literature you will start to develop an idea of the argument you are going to make about it. Decide if the paper is useful in supporting that wholeheartedly or whether it is adding extra points and nuances which will need building into the argument. Also consider, is this a better paper for supporting my argument than others I have read? In traditional reviews, some relevant papers will not make it into your final review if others are better examples.

Does it/could it refute my argument? It is not always necessary (despite common misconceptions), but sometimes it can be good to find a paper that disagrees with your developing argument. If this is particularly persuasive, you could change your mind and alter the idea you have for your overall argument. If you don't find it persuasive then it will be useful for showing you have considered other possible perspectives but then explain why you do not agree.

Does it/could it provide examples I can use? Can you use its methods (scientific/interviews/surveys etc) to back up your own choices? Can you use particular parts of its findings to support a particular part of your argument?

​Synthesising the literature

A review is more than a list of summaries of the papers you have found. Instead, you need to look for the connections and relationships between the sources. As you read, you are looking for: 

Using a synthesis matrix

To help with the process of sythesising you can use a table or a spreadsheet to organise your information and your thoughts. There are different ways to organise them but here are a few examples:

Thematic matrix

This is organised by theme - you may already have some in mind when you begin or you may just add more as you work. Add things when you think they may be a theme - you can always remove one if you realise it was actually only in one paper.

Source matrix

If you are doing a more systematic style review, then something like this may be more helpful:

Personal matrix

This is your review, so you can actually make your columns fit the needs of your topic or study. For example:

Synthesised writing

Once you have decided on themes, you need to make sure you are writing in a way that truly synthesises rather than lists papers under specific themes. Look at the following two examples:

Notice that in above example, the sentences and clauses begin with the authors, making them the subjects. This is typical of poor synthesis and your review becomes little more than he said/she said. The example below only has one sentence that starts with an author and that is a leading author in the field whose voice therefore carries more weight.

Another example: