Data driven decision making -Milestone 4

Surveys

In order to result in reliable, insightful data, surveys must be skillfully crafted. If you've ever taken a survey with confusing or poorly-worded questions, or one that was too long, you might reflect on how that experience likely made your response less useful to the researchers, just as it was unpleasant for you.

The Pew Research Center is a research institute primarily focused on developing public interest questionnaires. Review this article and the video below for an overview of the best practices in designing surveys and wording questions:

Demographic Data & Surveys

Many surveys choose to include a 'demographic information' section to elicit information about their subjects like race and ethnicity, income, or gender, that helps them to understand how responses vary across important sub-groups of their survey population. As much care needs to be devoted to the 'demographic information' section of your survey as the questions more directly related to your research questions:

1. Note that demographic categories change over time and by context. For example, the US Census has changed the options offered to respondents for selecting their race and ethnicity multiple times as this excellent infographic shows. See also this report about emerging approaches to Transgender and Alternative Gender Measurement in the 2018 General Social Survey.

2. Reflect on which demographic categories are of analytical interest for you given your research question, and ask only about those to reduce survey length and respondent risk of identification

3. Remember that demographic category identification has an element of subjectivity, and that respondent self-identification may not fit with the categories you have in mind for analyzing your survey. This is especially true of questions of economic class (most people identify as middle class, even when they are not) and ethnicity (where complex family histories and geographies are difficult to capture using single options for ethnic identification)

4. Consider respondent confidentiality; depending on the size of your respondent pool, asking comprehensive demographic questions might risk making confidential survey responses identifiable

Quantitative Data Analysis

As with qualitative research methods, quantitative researchers need to carefully design procedures that allow them to gather high-quality data, and procedures to analyze that data in order to answer their research questions. Quantitative research may gather their own data through surveys, experiments or observation, or make use of existing data that has been generated by other researchers, government, or interactions (as with research relying on the records produced through human interaction with the internet in the form of web traffic analytics).  We'll learn more about the best practices in survey design in the next sub-module; in this module you're going to explore how to evaluate the quality of data. 

When you encounter data, either used in a research study you're referencing or as you pursue your own original research, you need to ask some key questions in order to evaluate the value and suitability of that data for answering your research question. 

How was the data gathered/generated?

Whether you're working with primary source data sets or secondary sources that are working with data, high quality work will clearly identify the ways in which the data in question was gathered or generated.  Small changes in experimental procedures or survey outreach can have a big impact on your data.  Familiarize yourself with the genesis of the data set and ask yourself what biases might result from the procedures used to generate it.  High quality secondary sources should address any limitations in the gathering of their data or experimental methods frankly within the body of the paper. 

How are key concepts operationalized in this data?

Quantitative data sets reduce the complex social and natural phenomena of the world around us to analyzable numbers.  In the course of doing so, researchers have to craft variables which serve as proxies for those phenomena and allow us to group, count, correlate and differentiate that chaos.  Some phenomena are simple; your geographic location can be simplified into your zip code.  Others, like attitudes or aptitudes are a lot more complex, and are often represented by multiple variables.  Pay close attention to how the concepts your interested in have been translated into operationalized data, and compare the efforts of different researchers.  Ask yourself how well the variables correspond to the concepts you're interested in; what have they lost in translation?  How have the definitions shifted?  

Does this data really measure what I'm interested in?

Often researchers are forced to use more easily quantified measures as 'proxies' or stand-ins for unknown or difficult to capture phenomena.  For example, student's standardized scores are used as proxies for understanding academic aptitude or school quality.  As the video below explores, these scores might not be good proxies.  Similarly, ask yourself if the measure you're looking at in your data is really a good stand-in for the phenomena you're interested in; when looking at academic papers, observe how researchers discuss and justify any proxies they've used in their work and assess how they impact the applicability of that research to your questions. 

Is this data representative of the population/phenomena I want to study?

When conducting research about a subject for which there aren't any readily available data sets, sometimes we look to data about analogous populations or issues for insight.  When you're attempting to draw inferences from one group or context to another, be conscious of factors that could limit the applicability of that data.  A high-quality secondary source will discuss this issue if they've borrowed other data to apply to a different context; evaluate those explanations, and think critically about the limitations of any data you decide to transplant into your own research. 

Mixed Methods

As you've seen, qualitative and quantitative approaches to research both have advantages and disadvantages. Some researchers attempt to moderate the disadvantages by pursuing mixed methods approaches, where qualitative and quantitative elements are combined (Bazeley, 2018; Molina-Azorin, 2016). There are three main forms for mixed methods:

• Complementarity, when the results gathered from one method are elaborated or clarified with the results of another.

• Development, when the researcher uses one method to inform the development of the other methodological approach, as when using focus groups to develop quantitative surveys, or using economic data to select interview subjects.

• Expansion, when the researcher combines different methodological components for different elements of the study, extending the breadth of information in the study.

Note that it's not enough for a study to simply feature methods from both the qualitative and quantitative tool kits; what sets mixed methods apart is that the data from those methods is integrated to create knowledge greater than both methods considered separately.

To explore an example of mixed methods research in the real world, review this article from the Pew Research Center about the use of focus groups to enhance quantitative survey development, followed by this article about the use of quantitative data to enhance the development and analysis of focus groups.

Work Cited:

Albanese, M. (2022). Mixed Methods in Business, Management and Accounting Research: An Experimental Design in the Entrepreneurship Domain. European Journal of Interdisciplinary Studies, 8(1), 35–48. https://doi.org/10.26417/641eff87

Atske, S. (2020, December 18). How quantitative methods can supplement a qualitative approach when working with focus groups.Decoded. https://www.pewresearch.org/decoded/2020/12/18/how-quantitative-methods-can-supplement-a-qualitative-approach-when-working-with-focus-groups/

Bazeley, P. (2018). A Practical Introduction to Mixed Methods for Business and Management. Sage Publications Ltd.

Molina-Azorin, J. F. (2016). Mixed methods research: An opportunity to improve our studies and our research skills.European Journal of Management and Business Economics, 25(2), 37–38. https://doi.org/10.1016/j.redeen.2016.05.001

Silver, L. (2020, October 14). Testing survey questions ahead of time can help sharpen a poll’s focus.Pew Research Center: Decoded. https://medium.com/pew-research-center-decoded/testing-survey-questions-ahead-of-time-can-help-sharpen-a-polls-focus-cc46f80415f9

This basic page is a general-purpose layout that includes a banner.

Easy Editing Using the HTML Editor

You can use the HTML editor to make quick and easy changes without needing any knowledge of HTML. Enter your content and use the available controls to apply formatting to your text.

Changing the Banner

To Update the Banner: Make sure that the new image has dimensions similar to 1200 pixels in width x 300 pixels in height. To update the current banner enter the HTML editor, highlight the banner, and click the Insert Image icon. 

To Remove the Banner: Enter the HTML editor, highlight the banner, and use the <backspace>or <enter> key.

Copying Text

Pro Tip: When writing content, it is a great practice to first write content in a document, such as Microsoft Word. It allows stakeholders to easily collaborate and track changes to content. It also allows you to spot spelling and grammar errors early on.

When pasting text from a Word document into the HTML editor, however, some of the document’s text styling will copy over. This will clash with the styles that are carefully crafted for this template. So, we recommend pasting the text without formatting.

To paste text without formatting, you can use Ctrl+Shift+V (Cmd+Shift+V on Mac) to paste copied text as unformatted text to HTML editor.

For basic editing, please check Brightspace Help: Format HTML course content.this link opens in a new window/tab)