Data driven decision making -Milestone 2
Finding Reliable Sources
In the 21st century, there is a wide variety of information instantly available to those of us with an internet connection. That wealth of data comes with it's own problems: how do we efficiently and effectively find only the information that is relevant to our topic? How can we tell what information is of high quality?
To navigate this complexity, first we're going to divide data into three categories: Primary, secondary, and tertiary information. Each of these types plays a different role in your research process, so we'll need to apply slightly different standards to each depending on the use to which you're putting the data.
After that we're going to discuss how to effectively search for and evaluate academic research, reports, news media, general websites, and other media as a part of your research process.
If you haven't already done so, now is also a great time to get started with a citation manager. Citation managers are programs that plug into your browser and word processor to help you track, manage, and properly cite the research you're collecting. Their use isn't mandatory in this course, but it may make your life easier. See the Course Resources module for more information on Zotero and Mendeley, two free citation managers you can experiment with using.
Primary, Secondary & Tertiary Sources
Dividing information into primary, secondary, and tertiary sources is one way of characterizing different types of sources and the role that they play in research. Mastering these distinctions is valuable because it enables you to make strategic and thoughtful use of the information sources in these categories, while understanding the biases, limitations, and advantages of each.
Primary Sources:
Primary sources are the direct observations, research data, text of laws or other original documents, eye-witness accounts, artifacts, survey data, economic statistics and census results that constitute the raw material for research. Although these sources include a wide range of materials, they share that they were directly generated by the phenomena you're studying, without additional interpretation or analysis.
Primary sources, since they're not subject to the interpretation and analysis of other researchers, are an opportunity for you to apply your own lens and evaluation to your subject matter. Referencing primary sources whenever possible should be incorporated into your research practice since this is the raw material for original analysis, and allows you to more thoroughly evaluate the conclusions of other researchers who have referenced those same raw materials.
Secondary Sources
Secondary sources are the books, articles, essays, and documentaries that are one-step removed from primary source. Secondary sources are built on top of primary sources; they provide analysis and insight into a topic using primary sources for evidence, and will often quote from or otherwise reference primary sources.
Tertiary Sources
Tertiary sources are a further step removed from primary sources, and provide more general overviews on a topic. Tertiary sources are often textbooks, encyclopedia articles, dictionaries, anthologies, and other broad overviews of a topic. These sources can be valuable for conducting initial research to define your topic and narrow your research question, and can help you locate important primary and secondary sources for that topic, but otherwise will usually be too general to play a substantial role in your research.
Categorizing Sources & Context
The ways in which you are using a source will impact its role as a primary or secondary source. An article on the effectiveness of different executive leadership styles would be a secondary source if you were writing a paper about the relationship of leadership approaches to worker productivity. If, however, you were writing about how ideas about executive leadership have changed over time, that same article could serve as primary evidence about the ideas of the author.
Need more clarification? Here's some further reading:
Primary, Secondary, and Tertiary Sources | University of Minnesota Crookston: https://crk.umn.edu/library/primary-secondary-and-tertiary-sources
Library Guides: How to Research Guide: Primary, Secondary, & Tertiary: https://davenport.libguides.com/research-how/sources
this link opens in a new window/tab)
Searching Effectively for Souces
In this sub-module we're going to look at different types of primary, secondary, and tertiary sources and discuss how to effectively search for each.
Primary Sources:
Searching for and analyzing primary sources can be intimidating, and it can be tempting to rely on existing secondary sources instead. Primary source research is the bedrock of new discoveries, and is more accessible than you may think. Depending on your topic, there are many great online sources for accessing data, laws, reports, and other materials that will be valuable for your work.
When accessing data, make sure you reflect on it's context, quality, and how reputable the source is as you use it. Search for data that has the following characteristics:
Methodological Transparency: You are told how the data is gathered, and methodological choices made in producing the data set are shared.
Reputable Source: If the source of the data is clearly documented, and is reputable. In the case of smaller datasets from research projects or data set repositories, you are able to trace the source to it's author and evaluate whether the data is trustworthy.
Recency: Make sure that the data you're choosing to work with is sufficiently recent to support your argument.
Relevance & Specificity: The data should be sufficiently granular to give you insights into your area of interest. For instance, if you're interested in white-collar crime, national crime statistics won't necessarily provide you with much insight unless it differentiates the types of crimes tallied.
Below are some great resources to start searching for primary source data.
Economic Data
US Treasury Department: The US treasury department maintains a website featuring data and reports related to US fiscal trends.
US Department of Commerce: The Bureau of Economic Analysis, which is part of the Department of Commerce, has comprehensive data and analysis related to commerce, GDP, Income, International trade, etc.
United Staes Bureau of Labor Statistics: Data and analysis in the field of labor economics from the US government.
State of Maine: While you can find information about the state of Maine in the sources listed above, Maine (like most states) maintains its own repository of statistical and economic information.
International Labour Organization (ILO): The ILO is an agency of the Untied Nations focused on developing labour statistics and research about 187 member countries (including the United States).
World Bank: The World Bank is an International development agency whose mandate includes the production of international economic statistics and indicators.
Kaggle Data Sets: Kaggle is a data repository where data scientists post data scrapped and gathered from a range of sources, from daily stocks to product pricing to consumer behavior and census extracts. Be sure to carefully read the included information for every data set about how it was gathered, cleaned, and shaped, and how it can be used.
Survey & Census Data
US Census: The US census provides detailed data about workforce participation, economic trends, and other socio-demographic issues that may be of interest to students in this course.
General Social Survey: This survey has been run since 1972 and contains comprehensive data about attitudes, socio-demographic trends, and household-level economics. The data is freely accessible and can be used to understand how attitudes change over time, or differ between groups.
Pew Research Center: Pew is a public affairs-focused survey group, and many of their data sets have been made public.
Propublica
General Data Sources
Academic Data: Increasingly, academics are sharing the data that has been generated through their research to increase transparency and access (especially when it is publicly funded). Sites like Figshare, RE3DATA, OSF and the Harvard Dataverse, among others, host those research data sets for others to use.
US Open Data: This US government website cataolgues and links over 250k datasets related to a huge range of economic, social, health, governmental, geographic, and other areas of concern.
Qualitative Data Repository: This is home to interviews, documents, and other qualitative data (there's more information about qualitative data in the next module) generated by researchers engaged in qualitative research projects.
Secondary Sources:
When working with secondary sources, you will need to make sure that you are drawing predominantly from scholarly works. Scholarly work, as opposed to work that's written to appeal to a broader or popular audience, will tend to have the following characteristics:
The author will have clearly stated credentials, and usually an affiliation with an academic or research institution
The text is peer reviewed, if an article.
Contains a bibliography or works cited page
For more information about differentiating between scholarly and popular books and articles, visit the Purdue OWL's Evaluating Sources: Where to Begin.
The majority of the secondary sources you will use for this course will be academic journal articles.
Finding & Using Academic Journal Articles
There are a few advantages to using these sources:
Journal articles are typically published more quickly than books, allowing you access to more timely research.
Most journal articles are peer reviewed, giving some assurance as to the quality and originality of the research.
There are easily-accessible mechanisms for understanding how reputable a journal is.
Journal indices and databases (including scholar.google.com, Jstor and Proquest/Proquest One Business) make it easy to find journal articles that you can download or view online.
For tips on how to effectively search for journal articles, check out these tutorials:
Finding Journal Articles: Search Tips
How to find journal articles
Other Secondary Sources:
You can find books, including some e-books through the UMPI library. Evaluate books for recency, and focus on books that have been published by university presses and academic publishers if you are using the book for research support.
Government & Institutional Reports: These can sometimes also constitute primary sources, depending on the nature of the research you're doing. Many of the groups listed in our primary source section also publish reports analyzing the data they produce.
Think tanks and Policy Institutes: Think tanks and policy institutes are bodies of experts that research and write about issues related to the central focus of the think tank or its clients. Note that think tanks and policy institutes are generally more openly partisan or biased about issues than other sources (which doesn't mean other sources are without bias!)
Tertiary Sources:
As mentioned in the previous sub-module, tertiary sources have a role to play in your preliminary research, but will very rarely be appropriate sources for your research paper. They are also generally the easiest sources to find. For that reason, we are not going to discuss finding them in detail here.
Evaluating Information Sources
Now that you know how to find information, how can you evaluate its quality? In this sub-module we're going to discuss the markers of high-quality research sources.
Below you'll see a list of considerations for evaluating how you can make use of the sources you find in the course of your research (adapted from (Ash, n.d.; Burkhardt et al., 2010)
Purpose and intended audience:
What is the goal of this source?
Are they trying to inform, persuade, entertain, sell something, or a combination?
How might that purpose shape the information included and the conclusions of the author?
Who was this source intended for?
Look at the tone, language, citations, and structure of the document to give you clues as to the intended audience.
Is it scholarly or popular?
Specialized or broad?
Introductory or for expert readers?
Authority and credibility:
Who is the author? Is it a person or organization?
What are the author's qualifications, experience, and affiliations?
If an organization, what is that organization's mission or goals?
If a person, what are their organizational affiliations? Do they provide authority, and/or are they possible indicators of bias?
Who is the publisher?
For books, journals and other print media, is the publisher a university press or commercial publisher?
For websites, is this a personal website? Is it affiliated with a reputable organization? What can you tell about the goals of that organization?
Accuracy and reliability:
How has the information in the article been produced?
Are the methods sound?
How has the data been analyzed?
What are the possible sources of bias in the data used
Would you be able to verify any of the data used to support this source's argument, if the source uses data?
If the source relies on original research, like surveys or experimental data, has the researcher been clear about the methods used?
Do they provide information about the limitations and flaws with their data? It is good practice for researchers to be transparent about any potential issues with their data.
Are the source's claims well-referenced (ie, in a bibliography or footnotes)? Would you be able to visit those sources and verify the claims made?
Currency and timeliness:
Do you know when this source was published?
While this is usually clear for books and articles, websites often lack publication dates.
Has this source been published recently enough that it will provide insight into your research?
Even if researching a historical event or unchanging phenomena, you'll want to make sure you're including recent work to search for new theories and discoveries
Have major events happened since the source was published that might shift perspectives?
ie, articles about the impact of public health on business from before COVID-19.
Objectivity and bias:
Is the source focused on opinion or facts?
Does the source apeal to reason and logic, using evidence?
Is it making emotional appeals? Or relying on vague generalizations? These are markers of propaganda, which should not be included in your paper.
Is the analysis of data sound, and not unduly distorted by bias?
What was the source of funding and institutional support for this work, and could that funding call the objectivity of the source into question?
Is the work promoting a particular social, political, economic, or other advocacy agenda?
Note that none of these are absolute rules; different research questions and topics require different approaches. What is an absolute rule is the necessity for you to critically engage with and analyze your sources to understand their limitations and strengths, and use them in a way that is consistent with those limitations and strengths (and of course it will be much easier to produce a high-quality paper when you're drawing on hiqh-quality sources)
For instance, if you are working on a paper about what workplace safety regulations your employer should lobby congress about, you might find reports from the International Labour Organization, Unions, Business Lobbyists, a commerce-focused think tank and scholars from business, sociology and economics. All of those sources come with bias; they will all produce and interpret information differently according to their goals, cultures, practices, and training, and they will likely come to different conclusions. This is fine! Evaluate the sources by looking at their methods, thoroughness, thoughtfulness, and viewpoint, and use that information to understand what value the source can be for YOUR research. Part of the scholarly work you do will be in synthesizing and describing the use you make of these different sources to develop your argument.
To see this in action, pay particular attention to the 'literature review' or 'theory' sections of journal articles and books. How do the scholars talk about different sources? How have they characterized the limitations and strengths of other work, and how do they identify and build on gaps in the literature?
Additional Notes:
Some sources bring additional concerns and means to evaluate their credibility.
Evaluating Journal Article Quality:
Not all academic, peer-reviewed journals are equal in quality, and the rise of for-profit journals and open access means that you need to do a bit of work to make sure the paper you're accessing is from a high-quality source. Your first step comes from how you find the article: the journals you find through scholar.google.com, your library's e-journal search, or the databases you can access through the library will almost always be sufficiently reputable for use in your paper. If you have doubts about whether the journal you're accessing is reputable, especially if you found it using a general search engine, check the Annual Reviews data for that journal to see how it stands in the field, and google the journal title to see what the journal's practices are about peer review.
Evaluating Websites & Wikipedia:
There are additional concerns when using online sources - websites are easy to produce and change, and it can be especially difficult to track down the authors and intent. Read this article to learn about the importance of understanding the impact of search engine optimization on websites, domain extensions, and, when and how to use wikipedia: Evaluating Digital Sources
Works Cited:
Ash, K. (n.d.). Research Guides: External Analysis Research: 5. Evaluating Sources. Retrieved July 1, 2023, from https://researchguides.library.brocku.ca/external-analysis/evaluating-sources
Burkhardt, J. M., MacDonald, M. C., & Rathemacher, A. J. (2010). Teaching Information Literacy: 50 Standards-Based Exercises for College Students, (2nd Edition). American Library Association.
General Guidelines—Purdue OWL®—Purdue University. (n.d.). Retrieved July 1, 2023, from https://owl.purdue.edu/owl/research_and_citation/conducting_research/evaluating_sources_of_information/general_guidelines.html
Tawatao, C. (n.d.). Library Guides: FAQ: How do I know if my sources are credible/reliable? Retrieved July 1, 2023, from https://guides.lib.uw.edu/research/faq/reliable
