Evaluating Secondary Sources
By now, you've amassed a number of secondary sources for your research project, some of which you listed in your last writing assignment, the proposal, and the discussion board assignment on finding sources. How can you tell which ones are worth further study for your research report and inclusion in your annotated bibliography?
Researchers don't necessarily agree with one another, and you may find competing claims with competing evidence. After all, if all scholarly articles agreed and had one definitive answer, there'd be no need for research or scientific inquiry. Among the theories that address this idea is Thomas Kuhn's theory of the paradigm shift: "Kuhn argued that a scientific revolution is a noncumulative developmental episode in which an older paradigm is replaced in whole or in part by an incompatible new one. But the new paradigm cannot build on the preceding one. Rather, it can only supplant it, for 'the normal-scientific tradition that emerges from a scientific revolution is not only incompatible but actually incommensurable with that which has gone before.' Revolutions close with total victory for one of the two opposing camps" (http://www.des.emory.edu/mfp/Kuhnsnap.html).
An example of this appears in the work of the recent Nobel Prize winners Barry J. Marshall and J. Robert Warren for their discovery of "the bacterium Helicobacter pylori and its role in gastritis and peptic ulcer disease." Although a few scientists in the 1930s and 1940s had argued that a bacterium could cause ulcers, scientific thinking at the time held that no bacterium could survive in the strongly acidic environment of the stomach, and research was stymied. Through research (including a daring experiment in which one of the pair swallowed a container of liquid filled with Heliocobacter pylori to prove his point), Marshall and Warren were able to identify the process through which peptic ulcers emerged and thus develop a process through which they could be treated. This revolutionized the treatment of ulcers and effected Kuhn's paradigm shift in scientific thinking about ulcers.
1. What do experts say about the field?
The experts you know would be your professors in the field, major figures whose work you respect, your classmates, and you. Obviously professionals in the field will have more information about reliable sources than those just beginning in the profession (you and your classmates), but you should take seriously the recommendations you have from all whom you consider knowledgeable. You might also want to interview your professor about useful resources for your report.
- Class. Has the book, article, or resource been assigned for a class, or has another of the author's or research group's works been assigned or recommended?
- Professional Community.Have your professors or your classmates cited this work or work by this author in their own writing?
- Conferences.What kinds of work is being presented at recent conferences in the field? Each discipline has one or more major conferences each year; try reading through conference abstracts to get a sense of what current research is being produced. (Example: Society for Neuroscience, http://www.sfn.org/index.cfm?pagename=abstracts_ampublications). Most databases of abstracts are available in subscription format; you can find them at http://www.wsulibs.wsu.edu/griffin/indexes.htm.
- Reviews. Has the work been reviewed in a professional journal? Many disciplines produce a year-end survey of all the work published during the year; in American literature, for example, it's American Literary Scholarship, while British literature has The Year in English Studies. Reading such review essays can give you a good sense of what's important.
- Citations. Has this work or this author/research group been cited in other publications?
- Practitioners. In addition to academic scholarship about your discipline, if your field has a practical component (finance, marketing, engineering, social work), what do practitioners in the field believe to be important ideas?
2. What kind of publication published the article?
Scholarly journals are the major source for your research project, but not all periodicals would qualify as "scholarly." General interest magazines like Time or Newsweek would not be considered scholarly, nor would industry trade publications like Engineering News Record or The Chronicle of Higher Education. A site at Cornell (http://www.library.cornell.edu/olinuris/ref/research/skill20.html) can help you to decide whether a publication qualifies or not.
- Peer Review. Is the periodical peer-reviewed? Peer-reviewed journals ask scholars knowledgeable in the field to read and pass judgment on manuscripts before they are published.
These journals are the most impartial and authoritative in judging new research.
- Editorial Board. Who is on the editorial board? Can you recognized these as well-known scholars in the field? Their reputations serve as one way to vouch for the kinds of articles that the journal publishes.
- Journal Reputation. Is the periodical one of the main journals in its field, or does it publish articles in a specialized area? Both types can be equally authoritative, but looking at the differences in what is published in each can tell you something about mainstream and emerging theories.
- Press. Is the journal published by a university press or major scientific or professional publisher (e.g., Blackwell, Elsevier, Lawrence Erlbaum, etc.) or by an organization with a nonscientific agenda (Discovery Institute, Heritage Institute)?
3. How does the article measure up?
As you read the article, think about the following questions.
- Does the article establish a sense of context by addressing previously published work on the subject?
- Is the research cited current? Is it authoritative? A study that cites only a few sources that are less authoritative can undermine its credibility.
- Does the article clearly establish how its argument differs from previous work and contributes to an ongoing scholarly conversation about the subject?
- Does the argument make sense? Are the assertions sensible, and are they supported by the available evidence?
- Do you notice any logical fallacies, such as post hoc reasoning?
- If you're working in a scientific field, ask these questions:
- Does the research design conform to good practices within your field?
- Does the writer explain the research design sufficiently, including the method used for random sampling?
- Is the experiment replicable?
- Are the hypotheses testable?
- Do you see any flaws in the design, collection, or assessment of data? Are statistical measures normally used on this type of data missing, for example? Is the sample size large enough to yield statistically significant results?
- Does the article make a true contribution to your knowledge about the subject?
- Does the article as a whole make sense, given what you know about the subject?
4. How does the article fit in with other research sources you've gathered?
As you take notes to write the summary paragraph and critique paragraph on each article for your annoted bibliography, consider these questions:
- Do the articles you've gathered take a similar approach to the subject, or do they disagree?
- Do the articles give any evidence of underlying theories or perspectives that might affect the author's interpretation of events? For example, articles on economics might interpret data very differently depending on the economic theories espoused by the author.
- Does one article refute or critique another? If so, how will you determine which one is correct?
- How do all the articles fit into the discipline-wide "conversation" about the issue?