Critiquing Literature (Research) Articles for a Dissertation

          Boote and Beile (2005) wrote that pre-dissertation students need to learn the skills required for analyzing and synthesizing research in order to successfully assess, synthesize, and improve research.  Such skills also enable students to select a dynamic dissertation topic, and apply the most appropriate methods for collecting and analyzing data.  Boote and Beile (2005) explained that having the abilities to "build on the scholarship and research of those who have come before us" (p. 3) is known as generativity, which gives meaning, integrity, and sophistication to research efforts.           Boote and Beile (2005) provided guidelines for a literature review, which includes providing an overview of the study, demarcation of what the study is and is not, and rationalizes the stated decisions.  Boote and Beile (2005) wrote that a literature review should extend beyond Creswell's guidelines, and "serve a critical role in gatekeeping, policing, and leading to new productive work, rather than merely mirroring research in a field" (p. 7). 

A literature review should contain "standards such as consistency, parsimony, elegance, and fruitfulness" (Boote and Beile, 2005, p. 7).  Eight elements for a successful literature review include "topicality, comprehensiveness, breadth, exclusion, relevance, currency, availability, and authority" (Boote and Beile, 2005, p. 7).  Boote and Beile (2005) also developed a literature review rubric that assesses "coverage, synthesis, methodology, significance, and rhetoric" (p. 8).  Assembling the following toolbox for evaluating research provides a set of questions based upon prior academic experiences, and from leaders in the field.

Lauer (2004) wrote that policymakers read education research, and attempt to make a decision about whether to trust the results and conclusions, policymakers then need to question if the research should be used to influence education policy, and how to implement changes. Because state and/or local factors, including the cost of implementation, influence policy decisions, researchers must be held accountable for the "quality, coherence, applicability and educational significance of the research" reported.  Conducting an appropriate literature review creates the infrastructure to accountability.

Part I: Toolbox for Evaluating Research Articles

Certain components as noted by Lauer (2004) that represent high quality research espouse a few characteristics such as validity, connection to prior research, ethical standards, and peer review.  The skills required for pre-dissertation students when evaluating prior research permit students (and policymakers) to appraise the trustworthiness of the research being reviewed.  Using the Applied Quick Primer (Exhibit A), and answering the following questions aid in achieving a good appraisal of a research article:

‌‌‌1.      Does the research design match the research questions?

2.      Have acceptable technical standards been adopted during data collection and analyses? 

3.      Does the current study cumulatively add to the current knowledge base?

4.      Does the current study's construction arise from prior research studies and conclusions?

5.      Does the current study impart accepted rules for ethical research thus avoiding

         researcher bias?  

6.      Have the articles under review been peer-reviewed?

7.      Are the researcher's findings coherent, e.g., based upon a theory or conceptual framework?

8.      Have the researcher's findings been replicated, and representative of a body of research?

9.      Does the research study have external validity such that the findings of the study apply to the situation of interest?  (Situation of interest includes the setting, participants, program or treatment.)

10.   If a policy or practice is changed or adopted based on the research results, what difference, if any, will it make to education?  (A research study's educational significance is indicated by the effect size of a program or practice.  Additionally, a meta-analysis reflects the average effect size of several studies, and is a more informative tool to determine educational significance.)

11.   Do the researcher's conclusions infer potentially harmful effects?

         The Board of Regents of the University of Wisconsin System (2011) provided useful

guidelines for reviewing literature: 

12.    Does the Problem Statement address whose problem it is?

13.    What are the potential negative consequences if the study is never conducted?

14.    Is there clarity in the Purpose Statement? 

15.    Does the purpose statement flow directly from the problem statement? 

16.    Do the research questions address the study's purpose and problem? 

17.    Is the research method identified? 

18.    Does the research design (e.g., case study, phenomenology, grounded theory, causal-

 comparative, correlational, quasi-experimental, etc.) answer the research questions? 

19.    Are the variables/constructs and/or phenomenon/concept/idea identified? 

20.    Is the study's specific population identified, including an estimate of the number of participants who will serve as the sample, based upon a power analysis (quantitative/mixed  method) or on design conventions (qualitative)? 

21.    Is the geographic location of the study identified?

         Northcentral University (2012) encapsulates the required elements for a dissertation's  literature review:

22.    Is the literature review an orderly, cohesive, and well-sequenced narrative that relates the problem under investigation to a body of scholarly work?

23.    Does the literature review involve a critical appraisal and synthesis of the relevant published research, including critical appraisals of the research design and methods of key studies?

24.    Does the literature review provide a chronological viewpoint about the research topic with the majority of the literature reviewed sourced from scholarly, peer-reviewed work available in the previous five years?

25.    Does the literature review provide a plentiful number of references to enable impartiality to the study's topic, and provide readers with a wide-range of information about the importance and background of the project?

        Trochim (2006) wrote that reviewing literature requires an inquiry regarding validity. 

26.    Has the literature provided measurements that infer valid conclusions or samples that enable valid inferences?  If so, how is validity stated?

         Mesher (n.d.) wrote that "if an argument is found to be invalid, all judgment must be

suspended because, to be acceptable, an argument must be valid" (par. 1):

27.    Does the researcher's argument represent a valid (acceptable) form?  If so, how is it   

supported?  (Evaluate the content of its premises to assess truthfulness—

verified/justified—or falsehood.)

28.    Does the researcher's argument represent an invalid (unacceptable) form? If so, how?

29.    Are the claims verified or justified because they follow these three rules?  Explain.

         --the claims do not conflict with what one knows or understands as true;

         --the claims do not impose a belief or acceptance of unsupported claims conflicting

           with what one knows or understands as true;

--the claims support an appropriate element of proof.

30.    What do I want to learn from reading this article? 

         Little and Parker (2010) provided a host of questions useful for an article review:  

31.    Is the type of research descriptive (what is there or what do we see), comparative (are findings general or comparable to other elements), or analytical (how does it work or what is the mechanism)?

32.    What are the key points of the article?  Example. 

33.    Is there proof such as data supporting the article's conclusions?  Example. 

34.    Is there a superior degree of evidence, and any limitations noted in the research methodology? 

35.    What is important about the researcher's conclusions? 

36.    Does the research follow the steps of the research process in a logical manner?   

37.    Were the participants fully informed about the nature of the research?  

38.    Was the autonomy/confidentiality of the participants guaranteed? 

39.    Were the participants protected from harm? 

40.    Was ethical permission granted for the study? 

41.    Was the data gathering instrument described?  Was the instrument appropriate?  How was it developed?  Were reliability and validity testing undertaken and the results discussed? Was a pilot study undertaken?

42.    If a hypothesis was identified, was it supported? 

43.    Was a recommendation for further research made?

         Guidelines from Coughian, Cronin, & Ryan (2008, p. 739) provided an additional idea for critiquing research relative to the qualitative research manner:

44.    Were credibility, dependability, transferability and goodness discussed? 

         The Methodist Hospital Employee Intranet (2012) presented guidelines for critiquing a literature review for a mixed-methods research project although most guidelines have already been touched upon:

45.    Is there an integrated summary of the current knowledge base regarding the research problem, or does the literature review contain opinion or anecdotal articles without any synthesis or summary of the whole?

References:

 

Board of Regents of the University of Wisconsin System. (2011). The writer's handbook: Learn how to write a review of literature. Retrieved from http://writing.wisc.edu/Handbook/ReviewofLiterature.html

falseBoote, D.N. & Beile, P. (2005, August/September). Scholars before researchers: On the centrality of the dissertation literature review in research preparation. Educational Researcher, 34(6), 3-15. Retrieved from ProQuest.

Coughian, M., Cronin, P., & Ryan, F. (2008). Step-by-step guide to critiquing research. Part 1: quantitative research. Retrieved from http://lancashirecare.files.wordpress.com/2008/03/step-by-step-guide-to-criti-research-part-1-quantitative-reseawrch.pdf

Coughian, M., Cronin, P., & Ryan, F. (2008). Step-by-step guide to critiquing research. Part 2: qualitative research. Retrieved from http://lancashirecare.files.wordpress.com/2008/03/2007-step-by-step-guide-to-critiquing-research-part-2-qualitative-research.pdf

Lauer, P.A. (2004). How do I know if the research warrants policy changes? Retrieved from http://www.ecs.org/html/educationIssues/Research/primer/researchwarrants.asp 

Lauer, P.A. (2004). Research utility assessment guide. Retrieved from http://www.ecs.org/html/educationIssues/Research/primer/rubric.pdf

Little, J.W., & Parker, R. (2010). How to read a scientific paper. Retrieved from http://www.biochem.arizona.edu/classes/bioc568/papers.htm#reading

Mesher, D. (n.d.). Mission critical: San Jose University critical thinking web page. Retrieved from http://www.sjsu.edu/depts/itl/graphics/main.html  

Methodist Hospital Employee Intranet. (2012). Mixed-methods research critique template. Retrieved from http://methodistintranet.fasthealth.com/docs/mixed_crit_template_school.pdf

Northcentral University. (2012). 2011 Dissertation handbook. Retrieved from http://learners.ncu.edu/ncu_diss/default.aspx?attendance=Y

Suggestions for Good Academic Writing

As a doctoral learner, discovering professors' expectations for submitted assignments can only be achieved by a thorough knowledge of the university's and professors' writing rudiments.  Doctoral learners' pre-assessment of these rudiments requires an in-depth review of peer-reviewed articles, using references such as the American Psychological Association's (APA) Manual, and other reliable resources.  In addition, gaining and applying such knowledge when writing affords doctoral learners opportunities for introspection as well as intellectual and academic growth as degree requirements are further achieved.   

      This paper presents an analysis of (1) how academic professionals gain insight into good academic writing, (2) common mistakes by academic writers, (3) criticisms against poor academic writing, and (4) solutions for avoiding poor academic writing.  Analyzing and critiquing sources, which elaborate upon good writing mechanics as provided by the APA and other reliable references, creates a standardized across-the-board utility when academic writing is required and produced for evaluation.  Knowledge of common writing mistakes, and how to avoid such mistakes provides writers with the basic and advanced skills required for today's doctoral learners' writing activities.  

Academic Professionals' Identification of Good Academic Writing

           Completing years of post-secondary education, and gaining practice as an academic professional, does not necessarily enable academic professionals and students to have an ability to discern good academic writing.  However, if academic professionals and students supplement reading and writing skills by reference materials such as the 6^th Edition of the American Psychological Association's (APA) Manual (2010), the web sites of The Little Brown Handbook and The Online English Grammar Guide as well as from libraries, and other reliable sources, significant reading and writing expertise is gained.  Furthermore, academic professionals and students who seek resources appraised by peers gain assurance that the material presented is reliable.  However, it is of the utmost importance that when reviewing such material that one impose critical thinking skills, and not assume that what is written and peer-reviewed is necessarily correct.

           For example, Anonymous' (2010) composition recounts examples from the Bulwer-Lytton Fiction Contest, the brain-child of the English Department at San Jose State University sponsoring the contest, which is "a whimsical literary competition that challenges entrants to compose the opening sentence to the worst of all possible novels" (para 6).  While this article is locatable in the ProQuest database, the article is not peer-reviewed nor is Kelly's (1999) or Smith's (1999) articles according to ProQuest.  However, Kelly (1999) notes that Professor Sokal, a New York University physics professor, was "published in a respected journal, Social Text, (which) was thoroughly researched and footnoted, but was essentially nonsense" (para 3).  Professor Sokal (Kelly, 1999) purposely wrote his article as a test for the staff at Social Text to see if his thesis would be challenged, and remarkably, it was not.  While peer-reviewed articles offer academic professionals a means to evaluate good academic writing, academic professionals must frequently evaluate non-peer-reviewed student compositions, which requires the use of expertise attained using other means such as from the APA (2010) as previously mentioned.     

Common Writing Mistakes of Academic Writers

            Academic writers' common writing mistakes include attributes such as dense, "obscure, often incomprehensible writing" (Kelly, 1999, para 2), "jargon-filled" rhetoric (para 11), and "incomprehensible sentence structure" (Smith, 1999, para 1).  In addition, academic writers frequently avoid authenticity while writing because it "requires more courage and more hard work than being educated since it can only be achieved by standing alone from, and by becoming independent of, those surrounding communal voices and texts" (Badley, 2008, p. 364).  Badley (2008) also wrote that "less proficient writers tend to write syntheses by simply borrowing sentences and connecting information from each source, without elaboration and integration" (p. 422).  Badley's (2008) research demonstrated that "When considering top-level (composition) structure, paragraphing, topic sentences and use of conclusion, students’ pre-test written syntheses scarcely met the structural criteria of a good exposition" (p. 434 ). 

Criticisms of Poor Academic Writers

           Smith (1999) wrote that "scholars (are) increasingly making themselves irrelevant" (para 7).  In an attempt to fit ideological agendas, academic writers change the structure of words and sentences, which distorts meaning (Smith, 1999).  Kelly (1999) indicates that a "major factor in tolerance of bad writing has been a lack of hard-nosed editing by academic journals" (para 18).  More up-to-date evaluations of academic writing provides insight about whether improvements in academic writing prevail or not since 1999.  For example, Addison and McGee (2010) wrote that "college faculty are not adequately preparing students for required writing tasks in the private or government sector" (p. 164), and recommend that "an online peer-reviewed journal (be) established that allows for more timely and detailed reports than possible with print publications" (p. 171).  A preponderance for academic writers to make writing mistakes continues as they create "incomprehensible writing and factionalism, resulting even more in their diminishment and incoherence" (Smith, 1999, para 2).

Avoiding Poor Academic Writing

            The focus of good academic writing is well stated by James Cook University (2011): "Good writing is unambiguous. The reader does not have to make a choice about what the writer possibly means" (James Cook University, 2011, para 24), and ambiguousness is avoided by using concrete rather than abstract terms (Australia's James Cook University, 2011).  The James Cook University (2011) summarizes other core principles and characteristics for academic writing.  Core principles include a "reader-centred approach, outlining and drafting, a clear layout, logical presentation of material, pre-emption of reader’s questions, parallel structure, concise writing, precise details and information as well as good grammar, spelling and punctuation" (para 6).  Characteristics of academic writing include that "all statements can be supported by evidence; paragraphs have a single, but developed, theme; paragraphs begin with a theme sentence; (contain) neutral language; omit slang and jargon; avoid pronouns; (have a) judicious use of adjectives; (contain) precise information, verbs and word choice" (James Cook University, 2011, para 9). 

            Addison and McGee (2010) further indicated that college faculty identified "the five most important characteristics of good writing as…organization…analysis data/ideas/arguments, and use of supporting evidence" (p. 166).  Supporting the five characteristics is the systematic step-by-step writing process, which includes prewriting, higher order concerns, lower order concerns, and editing.  By learning and applying proven writing processes, poor academic writing is avoided.

Conclusion

            This paper presented an analysis of (1) how academic professionals gain insight into good academic writing, (2) common mistakes by academic writers, (3) criticisms against poor academic writing, and (4) solutions for avoiding poor academic writing.  Analyzing and critiquing sources, which elaborate upon good writing mechanics as provided by the APA and other reliable references, creates a standardized across-the-board utility when academic writing is required and produced for evaluation.  Knowledge of common writing mistakes, and how to avoid such mistakes provides writers with the basic and advanced skills required for today's doctoral learners' writing activities.  

References:

Addison, J., & McGee, S.J. (2010, September). Writing in high school/writing in college: Research trends and future directions. College Composition and Communication, (62)1, p. 147. Retrieved January 9, 2011, from ProQuest.
        
Anonymous (2010, April 3). Robert Rector: Bad writing gets its just reward. San Gabriel Valley Tribune. Retrieved January 8, 2011, from ProQuest.

American Psychological Association (2010). Publication manual of the American Psychological Association, 6^th Ed. Washington, D.C.: American Psychological Association.

Badley, G. (2008). Developing (authentic?) academic writers. Quality Assurance in Education, (16), 4, p. 363. Retrieved January 10, 2011, from ProQuest.

James Cook University (2011). What is effective academic writing? Retrieved January 9, 2011, from https://egrs.jcu.edu.au/workshops/

academic-writing-workshops/what-is-effective-academic-writing

 

Kelly, R. (1999, April 8). Bad blood over bad writing: Critics say US academic language has become so convoluted that it is largely incomprehensible to the point where argument is becoming impossible. The Irish Times, p. 15. Retrieved January 8, 2011, from Proquest.

 
Smith, D. (1999, March 7). Academic: When the writing is bad, ideas get lost. Winston - Salem Journal, p. 17. Retrieved January 8, 2011, from ProQuest

Higher Order Concerns in Academic Writing: Chronology, Order of Importance and Generality

Three Examples of Higher Order Concerns in Writing Academically

1. Chronology

Having been a victim of bullying at a country-based grade school during the 1970s, the subject of bullying continues to intrigue me as it increasingly manifests in today's society to the point of causing suicide. With the ever-increasing use of the social networks on the internet, and a mounting ability for most grade-school students to have a cell phone and personal computer, cyber-bullying is quickly becoming a typical phenomenon. Consequently, empowering grade school children and parents with tools to avoid cyber-bullying has become a common topic in the 21st century.

A recent report by the American Academy of Child & Adolescent Psychiatry (2008) revealed that almost fifty-percent of school children experience bullying, and that experiencing bullying affects academic performance, and the social and emotional development of school children. Empowering grade school children and parents with tools to avoid cyber-bullying requires research into the technical security aspects of e-communications such as provided by cell phones, computers, and other e-tools. While schools can prevent use of e-tools like cell phones and computers during school hours, students and parents carry the responsibility of after-school e-tool use.

2. Order of Importance

Empowering grade school children and parents with tools to avoid cyber-bullying has become an increasingly relevant topic of the 21st century. Having been a victim of bullying at a country-based grade school during the 1970s, the subject of bullying continues to intrigue me as it increasingly manifests in today's schools to the point of causing suicide. A recent report by the American Academy of Child & Adolescent Psychiatry (2008) revealed that almost fifty-percent of school children experience bullying, and that experiencing bullying affects academic performance, and the social and emotional development of school children. Empowering grade school children and parents with tools to avoid cyber-bullying requires research into the security features of e-communications such as provided by cell phones and other e-tools. While schools can prevent use of e-tools like cell phones and computers during school hours, students and parents carry the responsibility of after-school e-tool use.

3. Order of Generality

Having been a victim of verbal bullying at a country-based grade school during the 1970s, the subject of bullying continues to intrigue me as it increasingly manifests in today's schools to the point of causing suicide. A recent report by the American Academy of Child & Adolescent Psychiatry (2008) revealed that almost fifty-percent of grade school children experience bullying. Experiencing bullying affects academic performance, and the social and emotional development of school children (American Academy of Child & Adolescent Psychiatry, 2008).

Unfortunately, the widespread use of cell phones and social networking sites by grade school students to communicate with each other has manifested in additional avenues for transmitting abusive behavior. While schools can prevent the use of e-tools like cell phones and computers during school hours, students and parents carry the responsibility of after-school e-tool use. Empowering grade school children and parents with tools to avoid cyber-bullying requires research into the technical aspects of e-communications such as provided by cell phones and other e-tools. By providing students and parents with training on how to set up and implement the security features on cell phones and computers, abusive behaviors are avoidable.


References

American Academy of Child & Adolescent Psychiatry. (2008, May). Bullying: Facts for families (No. 80). Retrieved January 17, 2010, from http://www.aacap.org/galleries/ FactsForFamilies/80_bullying.pdf