Critiquing Literature (Research) Articles for a Dissertation

          Boote and Beile (2005) wrote that pre-dissertation students need to learn the skills required for analyzing and synthesizing research in order to successfully assess, synthesize, and improve research.  Such skills also enable students to select a dynamic dissertation topic, and apply the most appropriate methods for collecting and analyzing data.  Boote and Beile (2005) explained that having the abilities to "build on the scholarship and research of those who have come before us" (p. 3) is known as generativity, which gives meaning, integrity, and sophistication to research efforts.           Boote and Beile (2005) provided guidelines for a literature review, which includes providing an overview of the study, demarcation of what the study is and is not, and rationalizes the stated decisions.  Boote and Beile (2005) wrote that a literature review should extend beyond Creswell's guidelines, and "serve a critical role in gatekeeping, policing, and leading to new productive work, rather than merely mirroring research in a field" (p. 7). 

A literature review should contain "standards such as consistency, parsimony, elegance, and fruitfulness" (Boote and Beile, 2005, p. 7).  Eight elements for a successful literature review include "topicality, comprehensiveness, breadth, exclusion, relevance, currency, availability, and authority" (Boote and Beile, 2005, p. 7).  Boote and Beile (2005) also developed a literature review rubric that assesses "coverage, synthesis, methodology, significance, and rhetoric" (p. 8).  Assembling the following toolbox for evaluating research provides a set of questions based upon prior academic experiences, and from leaders in the field.

Lauer (2004) wrote that policymakers read education research, and attempt to make a decision about whether to trust the results and conclusions, policymakers then need to question if the research should be used to influence education policy, and how to implement changes. Because state and/or local factors, including the cost of implementation, influence policy decisions, researchers must be held accountable for the "quality, coherence, applicability and educational significance of the research" reported.  Conducting an appropriate literature review creates the infrastructure to accountability.

Part I: Toolbox for Evaluating Research Articles

Certain components as noted by Lauer (2004) that represent high quality research espouse a few characteristics such as validity, connection to prior research, ethical standards, and peer review.  The skills required for pre-dissertation students when evaluating prior research permit students (and policymakers) to appraise the trustworthiness of the research being reviewed.  Using the Applied Quick Primer (Exhibit A), and answering the following questions aid in achieving a good appraisal of a research article:

‌‌‌1.      Does the research design match the research questions?

2.      Have acceptable technical standards been adopted during data collection and analyses? 

3.      Does the current study cumulatively add to the current knowledge base?

4.      Does the current study's construction arise from prior research studies and conclusions?

5.      Does the current study impart accepted rules for ethical research thus avoiding

         researcher bias?  

6.      Have the articles under review been peer-reviewed?

7.      Are the researcher's findings coherent, e.g., based upon a theory or conceptual framework?

8.      Have the researcher's findings been replicated, and representative of a body of research?

9.      Does the research study have external validity such that the findings of the study apply to the situation of interest?  (Situation of interest includes the setting, participants, program or treatment.)

10.   If a policy or practice is changed or adopted based on the research results, what difference, if any, will it make to education?  (A research study's educational significance is indicated by the effect size of a program or practice.  Additionally, a meta-analysis reflects the average effect size of several studies, and is a more informative tool to determine educational significance.)

11.   Do the researcher's conclusions infer potentially harmful effects?

         The Board of Regents of the University of Wisconsin System (2011) provided useful

guidelines for reviewing literature: 

12.    Does the Problem Statement address whose problem it is?

13.    What are the potential negative consequences if the study is never conducted?

14.    Is there clarity in the Purpose Statement? 

15.    Does the purpose statement flow directly from the problem statement? 

16.    Do the research questions address the study's purpose and problem? 

17.    Is the research method identified? 

18.    Does the research design (e.g., case study, phenomenology, grounded theory, causal-

 comparative, correlational, quasi-experimental, etc.) answer the research questions? 

19.    Are the variables/constructs and/or phenomenon/concept/idea identified? 

20.    Is the study's specific population identified, including an estimate of the number of participants who will serve as the sample, based upon a power analysis (quantitative/mixed  method) or on design conventions (qualitative)? 

21.    Is the geographic location of the study identified?

         Northcentral University (2012) encapsulates the required elements for a dissertation's  literature review:

22.    Is the literature review an orderly, cohesive, and well-sequenced narrative that relates the problem under investigation to a body of scholarly work?

23.    Does the literature review involve a critical appraisal and synthesis of the relevant published research, including critical appraisals of the research design and methods of key studies?

24.    Does the literature review provide a chronological viewpoint about the research topic with the majority of the literature reviewed sourced from scholarly, peer-reviewed work available in the previous five years?

25.    Does the literature review provide a plentiful number of references to enable impartiality to the study's topic, and provide readers with a wide-range of information about the importance and background of the project?

        Trochim (2006) wrote that reviewing literature requires an inquiry regarding validity. 

26.    Has the literature provided measurements that infer valid conclusions or samples that enable valid inferences?  If so, how is validity stated?

         Mesher (n.d.) wrote that "if an argument is found to be invalid, all judgment must be

suspended because, to be acceptable, an argument must be valid" (par. 1):

27.    Does the researcher's argument represent a valid (acceptable) form?  If so, how is it   

supported?  (Evaluate the content of its premises to assess truthfulness—

verified/justified—or falsehood.)

28.    Does the researcher's argument represent an invalid (unacceptable) form? If so, how?

29.    Are the claims verified or justified because they follow these three rules?  Explain.

         --the claims do not conflict with what one knows or understands as true;

         --the claims do not impose a belief or acceptance of unsupported claims conflicting

           with what one knows or understands as true;

--the claims support an appropriate element of proof.

30.    What do I want to learn from reading this article? 

         Little and Parker (2010) provided a host of questions useful for an article review:  

31.    Is the type of research descriptive (what is there or what do we see), comparative (are findings general or comparable to other elements), or analytical (how does it work or what is the mechanism)?

32.    What are the key points of the article?  Example. 

33.    Is there proof such as data supporting the article's conclusions?  Example. 

34.    Is there a superior degree of evidence, and any limitations noted in the research methodology? 

35.    What is important about the researcher's conclusions? 

36.    Does the research follow the steps of the research process in a logical manner?   

37.    Were the participants fully informed about the nature of the research?  

38.    Was the autonomy/confidentiality of the participants guaranteed? 

39.    Were the participants protected from harm? 

40.    Was ethical permission granted for the study? 

41.    Was the data gathering instrument described?  Was the instrument appropriate?  How was it developed?  Were reliability and validity testing undertaken and the results discussed? Was a pilot study undertaken?

42.    If a hypothesis was identified, was it supported? 

43.    Was a recommendation for further research made?

         Guidelines from Coughian, Cronin, & Ryan (2008, p. 739) provided an additional idea for critiquing research relative to the qualitative research manner:

44.    Were credibility, dependability, transferability and goodness discussed? 

         The Methodist Hospital Employee Intranet (2012) presented guidelines for critiquing a literature review for a mixed-methods research project although most guidelines have already been touched upon:

45.    Is there an integrated summary of the current knowledge base regarding the research problem, or does the literature review contain opinion or anecdotal articles without any synthesis or summary of the whole?

References:

 

Board of Regents of the University of Wisconsin System. (2011). The writer's handbook: Learn how to write a review of literature. Retrieved from http://writing.wisc.edu/Handbook/ReviewofLiterature.html

falseBoote, D.N. & Beile, P. (2005, August/September). Scholars before researchers: On the centrality of the dissertation literature review in research preparation. Educational Researcher, 34(6), 3-15. Retrieved from ProQuest.

Coughian, M., Cronin, P., & Ryan, F. (2008). Step-by-step guide to critiquing research. Part 1: quantitative research. Retrieved from http://lancashirecare.files.wordpress.com/2008/03/step-by-step-guide-to-criti-research-part-1-quantitative-reseawrch.pdf

Coughian, M., Cronin, P., & Ryan, F. (2008). Step-by-step guide to critiquing research. Part 2: qualitative research. Retrieved from http://lancashirecare.files.wordpress.com/2008/03/2007-step-by-step-guide-to-critiquing-research-part-2-qualitative-research.pdf

Lauer, P.A. (2004). How do I know if the research warrants policy changes? Retrieved from http://www.ecs.org/html/educationIssues/Research/primer/researchwarrants.asp 

Lauer, P.A. (2004). Research utility assessment guide. Retrieved from http://www.ecs.org/html/educationIssues/Research/primer/rubric.pdf

Little, J.W., & Parker, R. (2010). How to read a scientific paper. Retrieved from http://www.biochem.arizona.edu/classes/bioc568/papers.htm#reading

Mesher, D. (n.d.). Mission critical: San Jose University critical thinking web page. Retrieved from http://www.sjsu.edu/depts/itl/graphics/main.html  

Methodist Hospital Employee Intranet. (2012). Mixed-methods research critique template. Retrieved from http://methodistintranet.fasthealth.com/docs/mixed_crit_template_school.pdf

Northcentral University. (2012). 2011 Dissertation handbook. Retrieved from http://learners.ncu.edu/ncu_diss/default.aspx?attendance=Y