EVALUATING SOURCES FOR RELEVANCE AND RELIABILITY

5.4 EVALUATING SOURCES FOR RELEVANCE
AND RELIABILITY
When you start looking for sources, you’ll find more than you can use, so
you must quickly evaluate their usefulness. To do so use two criteria:
relevance and reliability.
5.4.1 Evaluating Sources for Relevance
If your source is a book, do this:
• Skim its index for your keywords, then skim the pages on which those words
occur.
• Skim the first and last paragraphs in chapters that use a lot of your keywords.
• Skim prologues, introductions, summary chapters, and so on.
• Skim the last chapter, especially the first and last two or three pages.
• If the source is a collection of articles, skim the editor’s introduction.
• Check the bibliography for titles relevant to your topic.
If your source is an e-book, you should still follow these steps, but you can
also search the whole text for your keywords.
If your source is an article, do this:
• Read the abstract, if it has one.
• Skim the introduction and conclusion; or if they are not marked off by
headings, skim the first six or seven paragraphs and the last four or five.
• Skim for section headings, and read the first and last paragraphs of those
sections.
• Check the bibliography for titles relevant to your topic.
If your source is online, do this:
• If it looks like a printed article, follow the steps for a journal article, and also
search on your keywords.
• Skim sections labeled “introduction,” “overview,” “summary,” or the like. If
there are none, look for a link labeled “About the Site” or something similar.
• If the site has a link labeled “Site Map” or “Index,” check it for your
keywords and skim the referenced pages.
• If the site has a “search” resource, type in your keywords.
This kind of speedy reading can guide your own writing and revision. If
you do not structure your paper so your readers can skim it quickly and see
the outlines of your argument, your paper has a problem, an issue we discuss
in chapters 12 and 13.
5.4.2 Evaluating Sources for Reliability
You can’t judge a source until you read it, but there are signs of its reliability:
1. Is the source published or posted online by a reputable press? Most
university presses are reliable, especially if you recognize the name of the
university. Some commercial presses, which are presses not associated with a
university, are reliable in some fields, such as Norton in literature, Ablex in
sciences, or West in law. Be skeptical of a commercial book that makes
sensational claims, even if its author has a PhD after his name. Be especially
careful about sources on hotly contested social issues such as stem-cell
research, gun control, and global warming. Many books and articles are
published by individuals or organizations driven by ideology. Libraries often
include them for the sake of coverage, but don’t assume they are reliable.
2. Was the book or article peer-reviewed? Most reputable presses and journals
ask experts to review a book or article before it is published; this is called peer
review. Essay collections published by university presses are often but not
always peer-reviewed; sometimes they are reviewed only by the named editor
or editors. Few commercial magazines use peer review. If a publication hasn’t
been peer-reviewed, be suspicious.
3. Is the author a reputable scholar? This is hard to answer if you are new to a
field. Most publications cite an author’s academic credentials; you can find
more with a search engine. Most established scholars are reliable, but be
cautious if the topic is a contested social issue such as gun control or abortion.
Even reputable scholars can have axes to grind, especially if their research is
financially supported by a special interest group. Go online to check out
anyone an author thanks for support, including foundations that supported her
work.
4. If the source is available only online, is it sponsored by a reputable
organization? A website is only as reliable as its sponsor. You can usually
trust one that is sponsored and maintained by a reputable organization. But if
the site has not been updated recently, it may have been abandoned and may
no longer be endorsed by its sponsor. Some sites supported by individuals are
reliable; most are not. Do a web search for the name of the sponsor to find out
more about it.
5. Is the source current? You must use up-to-date sources, but what counts as
current depends on the field. In computer science, a journal article can be outof-
date in months; in the social sciences, ten years pushes the limit.
Publications have a longer shelf life in the humanities: literary or art criticism,
for example, can remain relevant for decades and even centuries. In general, a
source that sets out a major position or theory that other researchers accept
will stay current longer than those that respond to or develop it. Assume that
most textbooks are not current. If you are unsure whether a source will be
considered current, take your lead from the practice of established researchers
in the field. Look at the dates of articles in the works cited lists of a few recent
books or articles in the field: a good rule of thumb is that you can cite works
as old as the older ones in that list (but to be safe, perhaps not as old as the
oldest). Try to find a standard edition of primary works such as novels, plays,
letters, and so on: it is usually not the most recent. Be sure that you consult the
most recent edition of a secondary or tertiary source: researchers often change
their views, even rejecting ones they espoused in earlier editions.
6. If the source is a book, does it have notes and a bibliography? If not, be
suspicious, because you have no way to follow up on anything that the source
claims.
7. If the source is a website, does it include bibliographical data? You cannot
judge the reliability of a site that does not indicate who sponsors and
maintains it, who wrote what’s posted there, and when it was posted or last
updated.
8. If the source is a website, does it approach its topic judiciously? Your
readers are unlikely to trust a site that engages in heated advocacy, attacks
those who disagree, makes wild claims, uses abusive language, or makes
errors of spelling, punctuation, and grammar.
The following criteria are particularly important for advanced students:
9. If the source is a book, has it been well reviewed? Many fields have indexes
to published reviews that tell you how others evaluate a source (see our
“Appendix: Bibliographic Resources”).
10. Has the source been frequently cited by others? You can roughly estimate
how influential a source is by how often others cite it. Citation indexing makes
this easy to do (see 5.2.2). If you find that a source is cited repeatedly by other
scholars, you can infer that experts in the field regard it as reliable and
significant. Such sources are said to have a high “impact factor.” You should
keep an eye out for such sources and use them to orient yourself in your field
of research.
Whom Can You Trust?
The highly respected Journal of the American Medical Association appointed a
committee to review articles published by reputable journals for reliability.
Even though those papers had been approved by experts in the field, the
reviewers reported that “statistical and methodological errors were common”
(“When Peer Review Produces Unsound Science,” New York Times, June 11,
2002, p. D6). In the face of such revelations, some just dismiss what scientists
publish: if the reviewers of scientific articles can’t guarantee reliable data, what
is a mere layperson to do? You do what we all do—the best you can: read
critically, and when you report data, do so as accurately as you can. We’ll
return to this question in chapter 8.
Error is bad, but dishonesty is worse. One of Booth’s students got a summer
job with a drug company and was assigned to go through stacks of doctors’
answers to questionnaires and shred certain ones until nine out of ten of those
left endorsed the company’s product. These bogus data were then used to
“prove” that the product worked. The student quit in disgust and was, no doubt,
replaced by someone less ethical.
These indicators do not guarantee reliability. Reviewers sometimes
recommend that a reputable press publish something weakly argued or with
thin data because other aspects of its argument are too important to miss—we
have each done so. So don’t assume that you can read uncritically just
because a report is written by a reputable researcher and published by a
reputable press.