A literature review should contain "standards such as consistency, parsimony, elegance, and fruitfulness" (Boote and Beile, 2005, p. 7). Eight elements for a successful literature review include "topicality, comprehensiveness, breadth, exclusion, relevance, currency, availability, and authority" (Boote and Beile, 2005, p. 7). Boote and Beile (2005) also developed a literature review rubric that assesses "coverage, synthesis, methodology, significance, and rhetoric" (p. 8). Assembling the following toolbox for evaluating research provides a set of questions based upon prior academic experiences, and from leaders in the field.
Lauer (2004) wrote that policymakers read education research, and attempt to make a decision about whether to trust the results and conclusions, policymakers then need to question if the research should be used to influence education policy, and how to implement changes. Because state and/or local factors, including the cost of implementation, influence policy decisions, researchers must be held accountable for the "quality, coherence, applicability and educational significance of the research" reported. Conducting an appropriate literature review creates the infrastructure to accountability.
Part I: Toolbox for Evaluating Research Articles
Certain components as noted by Lauer
(2004) that represent high quality research espouse a few characteristics
such as validity, connection to prior research, ethical standards, and peer review. The skills required for pre-dissertation
students when evaluating prior research permit students (and policymakers) to appraise the trustworthiness
of the research being reviewed. Using the Applied Quick Primer (Exhibit
A), and answering the following questions aid in achieving
a good appraisal of a research article:
1. Does the research design match the research questions?
2. Have acceptable technical standards been
adopted during data collection and analyses?
3. Does the current study cumulatively add to
the current knowledge base?
4. Does the current study's construction
arise from prior research studies and conclusions?
5. Does the current study impart accepted
rules for ethical research thus avoiding
researcher bias?
8. Have
the researcher's findings been replicated, and representative of a body of
research?
9. Does
the research study have external validity such that the findings of the study
apply to the situation
of interest? (Situation of interest includes the setting,
participants, program or treatment.)
10. If
a policy or practice is changed or adopted based on the research results, what
difference, if any, will
it make to education? (A research
study's educational significance is indicated by the effect
size of a program or practice.
Additionally, a meta-analysis reflects the average effect
size of several studies, and is a more informative tool to determine educational
significance.)
11. Do the
researcher's conclusions infer potentially harmful effects?
guidelines for reviewing literature:
12. Does the Problem
Statement address whose problem it is?
13. What
are the potential negative consequences if the study is never conducted?
14. Is there clarity
in the Purpose Statement?
15. Does
the purpose statement flow directly from the problem statement?
16. Do
the research questions address the study's purpose and problem?
17. Is
the research method identified?
18. Does
the research design (e.g., case study, phenomenology, grounded theory, causal-
comparative,
correlational, quasi-experimental, etc.) answer the research questions?
19. Are
the variables/constructs and/or phenomenon/concept/idea identified?
20. Is
the study's specific population identified, including an estimate of the number
of participants who will serve as the sample, based upon a power analysis (quantitative/mixed method)
or on design conventions (qualitative)?
21. Is
the geographic location of the study identified?
Northcentral University (2012)
encapsulates the required elements for a dissertation's literature review:
22. Is the literature review an orderly,
cohesive, and well-sequenced narrative that relates the problem under investigation to a body of scholarly work?
23. Does the literature review involve a critical
appraisal and synthesis of the relevant published research, including critical appraisals of the research design and methods of
key studies?
24. Does
the literature review provide a chronological viewpoint about the research
topic with the
majority of the literature reviewed sourced from scholarly, peer-reviewed work available
in the previous five years?
25. Does
the literature review provide a plentiful number of references to enable
impartiality to the
study's topic, and provide readers with a wide-range of information about the importance
and background of the project?
Trochim (2006) wrote that reviewing literature
requires an inquiry regarding validity.
26. Has the literature provided measurements that
infer valid conclusions or samples that enable valid inferences? If so, how is validity stated?
Mesher (n.d.) wrote that "if an
argument is found to be invalid,
all judgment must be
suspended
because, to be acceptable, an argument must
be valid" (par. 1):
27. Does the researcher's argument represent a
valid (acceptable) form? If so, how is
it
supported? (Evaluate the content of its premises to
assess truthfulness—
verified/justified—or falsehood.)
28. Does the researcher's argument represent an
invalid (unacceptable) form? If so, how?
29. Are the claims verified or justified because
they follow these three rules? Explain.
--the
claims do not conflict with what one knows or understands as true;
--the
claims do not impose a belief or acceptance of unsupported claims conflicting
with what one knows or understands as true;
--the
claims support an appropriate element of proof.
30. What do I want to learn from reading
this article?
Little
and Parker (2010) provided a host of questions useful for an article review:
31. Is the type of research descriptive (what is there or what do we
see), comparative (are findings
general or comparable to other elements), or analytical (how does it work or
what is
the mechanism)?
32. What are
the key points of the article? Example.
33. Is there
proof such as data supporting the article's conclusions? Example.
34. Is there
a superior degree of evidence, and any limitations noted in the research methodology?
35. What is
important about the researcher's conclusions?
36. Does the research follow the steps of
the research process in a logical manner?
37. Were the
participants fully informed about the nature of the research?
38. Was the
autonomy/confidentiality of the participants guaranteed?
39. Were the participants
protected from harm?
40. Was ethical permission
granted for the study?
41. Was the data
gathering instrument described? Was the instrument
appropriate? How was it developed? Were reliability and validity testing
undertaken and the results discussed? Was a pilot study undertaken?
42. If a hypothesis
was identified, was it supported?
43. Was a recommendation
for further research made?
Guidelines from Coughian, Cronin, &
Ryan (2008, p. 739) provided an additional idea for critiquing
research relative to the qualitative research manner:
44. Were credibility, dependability,
transferability and goodness discussed?
The Methodist Hospital Employee
Intranet (2012) presented guidelines for critiquing a literature review for a mixed-methods research project
although most guidelines have already been touched upon:
45. Is there an integrated summary of the current knowledge base regarding
the research problem, or does the literature review
contain opinion or anecdotal articles without any synthesis or summary of the whole?
Board of Regents of the
University of Wisconsin System. (2011). The
writer's handbook: Learn how to write a
review of literature. Retrieved from http://writing.wisc.edu/Handbook/ReviewofLiterature.html
Boote,
D.N. &
Beile, P. (2005, August/September). Scholars before researchers: On the centrality of the dissertation literature review in research preparation. Educational Researcher, 34(6), 3-15. Retrieved from ProQuest.
Coughian, M., Cronin, P., & Ryan, F.
(2008). Step-by-step guide to critiquing
research. Part 1: quantitative
research. Retrieved from http://lancashirecare.files.wordpress.com/2008/03/step-by-step-guide-to-criti-research-part-1-quantitative-reseawrch.pdf
Coughian, M., Cronin, P., & Ryan, F.
(2008). Step-by-step guide to critiquing
research. Part 2: qualitative
research. Retrieved from http://lancashirecare.files.wordpress.com/2008/03/2007-step-by-step-guide-to-critiquing-research-part-2-qualitative-research.pdf
Lauer, P.A. (2004). How
do I know if the research warrants policy changes? Retrieved from http://www.ecs.org/html/educationIssues/Research/primer/researchwarrants.asp
Lauer, P.A. (2004). Research utility
assessment guide. Retrieved from http://www.ecs.org/html/educationIssues/Research/primer/rubric.pdf
Little, J.W., & Parker, R.
(2010). How to read a scientific paper. Retrieved from http://www.biochem.arizona.edu/classes/bioc568/papers.htm#reading
Mesher,
D. (n.d.). Mission critical: San Jose University critical thinking web page.
Retrieved from http://www.sjsu.edu/depts/itl/graphics/main.html
Methodist Hospital Employee Intranet.
(2012). Mixed-methods research critique
template. Retrieved
from http://methodistintranet.fasthealth.com/docs/mixed_crit_template_school.pdf
Northcentral University. (2012). 2011 Dissertation handbook. Retrieved from http://learners.ncu.edu/ncu_diss/default.aspx?attendance=Y
