Getting the Balance Right for Qualitative Data Collection
The last sweet days of summer are stretching out long in the orchard, and you are, once again, Simon’s fruit grower, inspecting your harvest. You have a barn overflowing with all the fruits you gathered using your chosen sampling method. What does this bounty tell you about the orchard?
Or, in other words, data collection is drawing to a close and you are faced with the job of turning all those words (or audio, video, imagery)— all that qualitative data into a concise, considered report.
You need to make decisions about your approach to qualitative data analysis. Those decisions sit on a spectrum of rigour, from formal and structured down one end, to informal and unstructured at the other.
Selecting the right approach and the right amount of rigour for your analysis is important for several big reasons.
There is no ‘correct’ amount of rigour, or no ‘sweet spot’ on the spectrum. It’s dependant on the project. What’s important is a good match between your project and the amount of rigour you apply. Under-engineer your approach and you risk an analysis that is not credible, reliable, transparent or ethical. Over-engineer your analysis and you risk frustrating your team and your client, taking up resources that could have been used to do something more beneficial.
To help you make these decisions, let’s explore what the options are for qualitative analysis and factors you might consider in selecting an appropriate approach.
Coding as qualitative analysis
While there are many forms of qualitative data analysis, one of the most common methods is coding. Coding refers to the process of systematically organising your data into retrievable buckets.
When discussing theoretical approaches to coding, there is an important distinction to make between inductive and deductive coding. Inductive coding (also known as ground up coding) is an approach where you derive your codes from the data. You don’t start with preconceived notions of what the codes should be, but rather, allow the narrative or theory to emerge from the data itself. This is useful for exploratory research or instances when you want to come up with a new theory, idea or concepts.
On the other hand, deductive coding (or top-down coding) is an approach where you start by developing a framework with your initial set of codes. This set might be based on your research questions, KEQs, interview guides of an existing framework or theory. You then read through the data and assign excerpts to predefined codes. This is good for when you have a pre-determined structure for how you need our final findings to be. For example, you and the client have agreed on a report structure based on a set of key evaluation questions. This is not a strict binary. It’s common to move between the two and take elements from each when coding in practice.
Image source: Delve Tool
Tools and theoretical approaches
There are many tools you can use to code data— NVivo, Excel and Dedoose are some examples of commonly used platforms. There are also many different methods and theoretical approaches to coding, including but not limited to:
- In Vivo coding: coding an excerpt of text based on a participant’s own words, rather than the researcher’s interpretation.
- Structural coding: coding your data according to research questions of topics such that you iteratively turn larger sets of semi-structured data (such as a collection of interviews with multiple stakeholders) into smaller pieces for further analysis.
- Values coding: coding excerpts that pertain to the participant’s values, attitudes, and beliefs.
- Simultaneous coding: method where a single excerpt of qualitative data is coded with multiple codes.
Choosing an approach
Given the diversity of approaches to coding and qualitative data analysis, how do we decide which approach to use, and when?
The simple answer to these questions is: it depends. When it comes to rigour in qualitative analysis, there are no hard and fast rules. Rather, it is a decision-making process where decisions must be made in the context of a particular project.[1] There are numerous factors which may influence how this decision is made, including, but certainly not limited to:
- The scale of the project – How many interviews are you planning to do? What is the length of the evaluation (multiple years vs. a couple months turnaround)? Are there multiple data collection periods or just one? Are you working across many jurisdictions? What is the complexity of the stakeholders you are engaging with and are they likely to hold different perspectives?
- Resourcing – What is your budget and how much money do you have to invest in the rigour of your qualitative analysis? How big is your team? How many people will be involved in doing interviews? How experienced is your team in different types of qualitative analysis and what expertise do you have?
- The complexity of the project – Are you asking about a sensitive subject matter? Are you going to need a lot of nuance? What are your client’s expectations about the level of rigour required? Will the findings from this evaluation be used for formal decision-making processes?
A rigorous approach to qualitative data means having a considered, intentional and theory-informed approach. This means applying the analytical approach appropriately to the task at hand.
Having intentional conversations with your team at the outset of the analysis process can enable you to carefully weigh up these factors and make a critical and well-informed decision about the level of rigour appropriate for the specific project.
Patterns and puzzles – a note about nuance
The point of a rigorous approach to qualitative data collection is not just to assign a wide spectrum of complex views to 11 tidy boxes. We are looking for both patterns (things that fit neatly in our boxes) and puzzles (things that don’t).
Bob Williams[2] shared four types of patterns and puzzles:
- Generalisations and exceptions (usually … but …)
- Contradictions (on the one hand … on the other hand …)
- Surprises (I’d expected … but….) (I didn’t expect … but ….)
- Puzzles (Why…?)
Sometimes it can be tempting to stop at generalisations and key themes. But this is the tip of the iceberg if we don’t also think about the exceptions and the contradictions. It’s about what people tell us, and what they don’t tell us. The particular words they select. The dominant view, yes, but also the minority perspectives that highlight the nuance we treasure in qualitative data analysis.
There are many considerations to selecting the appropriate approach to qualitative data analysis, but it’s worth always keeping in mind that your approach should help you find the patterns without obscuring the puzzles.
To take you back to the orchard (and risk over-extending a metaphor)— there is no ready recipe for turning your fruit into a salad. Getting the right level of rigour is dependant on a multitude of considerations, each unique to your project and highly interactive with each other.
The best way to approach the question of rigour in qualitative data analysis is to come back to basics. How do the aspects of credibility, reliability, transparency, bias, ethics and reflexivity apply to your project? Which if the considerations are relevant to your project? Thinking this through will most likely make it clear what the right approach is for your project. Now you just have to do it!
References
[1] Elliott, V., 2018. Thinking about the coding process in qualitative data analysis. Qualitative report, 23(11).
[2] Bob Williams (n.d.) Qualitative data analysis. Retrieved March 2024, from https://www.bobwilliams.co.nz/ewExternalFiles/analysis2.pdf
Kia ora to Nan Wehipeihana for sharing this reference.
