Category: data generation

Obtaining the data you really need: on conducting qualitative interviews

Leave a comment

I have been planning a new qualitative research project, and reading draft proposals and draft methodology chapters for students I am coaching, so I have been thinking about qualitative data lately; particularly how to get the right kinds of data from participants when we are conducting interviews.

There are three main forms of interview that are discussed in methodology texts and guides: structured, semi-structured and unstructured. Generally, when embarking on a qualitative project, students tend to opt for semi-structured interviews. This enables them to have a set series of questions, hopefully well connected to their theory and literature, but also to create space for participants, or interviewees, to add views and insights that may not strictly follow the questions. It’s sort of a best-of-both-worlds scenario. Unstructured interviews are hard to manage, especially for postgraduate students, many of who are doing this form of fieldwork for the first time. And fully structured interviews can veer into not allowing any space for additional, unpredicted insights and information – they can, in other words, limit the conversation, and also the kind of data collected. *Caveat: you do need to choose the right tool for your project, regardless of whether it is hard to do or not.

But, semi-structured interviews, while they seem to be a dominant preference for many students doing qualitative research via interviews, are not easy to do well. A key issue I am thinking about in relation to my new project, and that I assist other researchers with, is this: how do I conduct the actual interview so that I get the data I really need? What tools or techniques so I need to be aware of? I have listed a few points here that have helped me, and that are also the product of my own mistakes and learning in conducting qualitative interviews.

interview 1

Photo by nappy from Pexels

  • The point of the first interview is to get a second interview. 

I didn’t know this during my own PhD, which was also my first major research project. I did one set of interviews, and I do recall, on reading the transcripts, wishing that I could go back and ask some of the questions differently or find out more about specific issues. But I didn’t really know I could (and I didn’t plan for this so I ran out of time). But, now it makes complete sense. Unless you are interviewing someone you already know well, chances are you and the interviewee will be strangers to one another. Establishing a rapport, and trust, can take time – certainly more time than one interview allows for. So, you need to make the goal of the first interview establishing a connection that will enable you to go back, and talk to that person again. You may get the basic data you need in the first interview, but chances are that you will need more that interview #1 can yield. You will want richer, or deeper responses to some of the questions, or will see that certain questions could be asked differently, to yield slightly better, or more relevant, responses considering your research questions and aims.

Plan enough time for at least two interviews, and ensure that you check with the interviewee that you could approach them again if needed. Listen to the recording of the interview as soon as you can, then, make notes, and think about what the data is saying in relation to those research questions. Then work out whether you have enough of the rights kinds of data to include in your analysis, where any gaps are, and plan interview #2 accordingly.

  • Record your interviews and take notes.

You always want to audio-record the interviews, so that your transcription, analysis and reporting will be accurate, and rich. But, you also want to take notes – not so many that you are unable to maintain eye contact and engagement in the interview, but have a notebook and pen to record things that the audio may not capture. Perhaps body language is a factor to consider in terms of the kind of data you are generating – if participants appear nervous, or distracted, this may be something you want to consider in the analysis alongside their words. Perhaps what their words say, and what their body language says are two different things, and this could be important as a possible finding. Perhaps they mention names of other people you could consider interviewing as well, or names of websites, documents and other sources of information you need to follow up on. All of this can be captured in a few short notes, and can add to the richness of your overall findings and analysis later on.

  • Plan for a pilot interview.

This has been a big point of learning for me. I tend to get over-eager in interviews, and I just plain talk too much. I find myself listening to the audio and cringing, and wishing I could just be quiet, and let the interviewee talk! There are two benefits, here, to a pilot interview: the first is that you can practise being an interviewer. You can work out how to record and take notes, how to talk less and listen more, how to direct and redirect the conversation as needed, and how to manage your own facial expressions and body language, to remain as neutral as possible so as not to bias or shift the interview in unhelpful directions (like looking shocked or disapproving at something an interviewee says to you). The second is that you can figure out whether the style you are using and questions you have as a guide are yielding the answers you need. Are there too many questions? Are some of the questions too long and ‘wordy’? Do any of the questions yield answers that seem a bit off topic, or that are repetitive? Make notes and adjust the interview plan before the real interviews begin. You need to do a pilot with someone who represents the interviewee demographic, and preferably with enough time to make changes and adjustments before you start meeting with the study’s group of participants.

interview 2

Photo by nappy from Pexels

  • Plan for a conversation, rather than a Q&A session.

This has been a really big point of learning for me. In a qualitative study, regardless of the topic, you need rich, detailed data. Qualitative research is about depth, rather than breadth, in simple terms. Thus, you need depth in the data, to achieve depth in your analysis, findings and conclusions. Thus, what you want to do with conducting the interview is create a conversational space, rather than a stiff, Q&A session where you ask a long list of questions, and the interviewee responds. The best way, I find, is to start off by asking the interviewees to tell me a story, related to the research I am doing and my interview questions. For example, if I am doing a project on postgraduate supervision, and my interviewees are students and supervisors, I don’t want to start off with something like: “Do you enjoy supervising students, and why or why not?” (Typical kind of question you find in interview schedules). This will not help me to get a richer sense of what the supervisor does as a supervisor, and what aspects they do and do not enjoy. So, what I could rather do is ask the question like this: “Can you start off by telling me a bit about your current supervision situation?” And then just listen. In that response, then, I listen for what aspects they seem to like, or not like, and in the follow up, I could ask: “You seem to find supervision a struggle, rather than a joy, right now. Would you say that is accurate? Could you say a bit more about that?” And then listen. And so on. So my questions become a guide, but not a determinant to the structure of the interview.

The main thing in doing qualitative interviews, I am learning, is for an interviewer to have empathy. Trying to be in the moment, and create a sense for the interviewee that their stories are valuable, and worth sharing and hearing, enables the creation of a rapport that can lead to follow-on interviews, and that can encourage interviewees to see me as an ally, and someone who will share their stories responsibly, ethically and with care. This is crucial when you are working with people who trust you enough to give you truthful stories, experiences and accounts. Ultimately, you need to listen closely, and record accurately, so that what you share in your study is able to meaningfully shape knowledge, research and practice in your field.

Developing well-constructed data gathering tools, or methods, for your study

1 Comment

I spent the better part of last week working with emerging researchers who are at the stage of their PhD work where they are either working out what data they will need and how to get it, or sitting with all their data and working out how to make sense of it. So, we are talking theory, literature, methodology, analysis, meaning making, and also planning. In this post I want to focus on planning your data gathering phase, specifically developing ‘instruments’, such as questionnaires, interview schedules and so on.

tools

Whether your proposed study is quantitative, qualitative or mixed methods,  you will need some kind of data to base your thesis argument on. Examples may include data gathered from documents in the media, in archives, or from official sources; interviews and/or focus groups; statistical datasets; or surveys. Whatever data your research question tells you to generate, so as to find an answer, you need to think very carefully about how your theory and literature can be drawn into developing the instruments you will use to generate or gather this data.

In a lot of the postgraduate writing I have read and given feedback on, there are two main trends I have noticed in the development of research methods. The first is what I considered ‘too much theory’, and the other ‘not quite enough’. In the first instance, this is seen in researchers putting technical or conceptual terms into their interview questions, and actually asking the research questions in the survey form or interview schedule. For example: ‘Do you think that X political party believes in principle of non-racialism?’ Firstly, this was the overall research question, more or less. Secondly, this researcher wanted to interview students on campus, and needed to seriously think about whether this question would yield any useful data  – would her participants know what she meant by ‘the principle of non-racialism’ as she understood it theoretically, or even have the relevant contextual knowledge? Let’s unpack this a bit, before moving on to trend #2.

The first issue here is that you are not a reporter, you are a researcher. This means you are theorising and abstracting from your data to find an answer that has significance beyond your case study or examples. Your research questions are thus developed out of a deep engagement with relevant research and theory in your field that enables you to see both the ‘bigger picture’ as well as your specific piece of it. If you ask people to answer your research question, without a shared understanding of the technical/conceptual/theoretical terms and their meanings, you may well end up conflating their versions of these with your own, reporting on what they say as being a kind of ‘truth’, rather than trying to elicit, through theorising, valid, robust and substantiated answers to your research questions, using their input.

This connects to the second issue: it is your job to answer your research question, and it is your participants’ job to tell you what they know about relevant or related issues that reference your research question. For example, if you want to know what kinds of knowledge need to be part of an inclusive curriculum, you don’t ask this exact question in interviews with lecturers. Rather, you need to try and find out the answer by asking them to share their curriculum design process with you, talk you through how they decide what to include and exclude, ask them about their views on student learning, and university culture, and the role of the curriculum, and knowledge, in education. This rich data will give you far more with which to find an answer to that question than asking it right out could. You ask around your research questions, using theory and literature to help you devise sensible, accessible and research-relevant questions. This also goes for criteria for selecting and collating documents to research, should you be doing a study that does not involve people directly.

analysis of data

Photo by Startup Stock Photos from Pexels

The second trend is ‘not enough theory’. This tends to take the form of having theory that indicates a certain approach to generating data, yet not using or evidencing this theory in your research instruments.  For example structuralist theories would require you finding out what kinds of structures lie beneath the surface of everyday life and events, and also perhaps how they shape people, events and so on. An example of disconnected interview questions could be asking people whether they enjoy working in their university, and whether there are any issues they feel could be addressed and why, and what their ideal job conditions would be, etc., rather than using the theoretical insights to focus, for example, on how they experience doing research and teaching, and what kinds of support they get from their department, and what kinds of support they feel they need and where that does and should come from, etc. You need to come back to using the theory to make sense of your data, through analysis, so you need to ensure that you use the theory to help you create clear, unambiguous, focused questions that will get your participants, or documents, talking to you about what matters to your study. Disconnecting the research instruments from your theory, and from the point of the research, may lead to a frustrating analysis process where the data will be too ‘thin’ or off point to really enable a rich analysis.

Data gathering tools, or methods for getting the data you need to answer your research questions, is a crucial part of a postgraduate research study. Our data gives us a slice of the bigger research body we are connecting our study to, and enables us to say something about a larger phenomenon or set of meanings that can push collective knowledge forward, or challenge existing knowledge. This is where we make a significant part of our overall contribution to knowledge, so it is really important to see these instruments, or methods, not as technical or arbitrary requirements for some ethics committee. Rather, we need to conceptualise them as tools for putting our methodology into action, informed and guided by both the literature our study is situated within as well as what counts as our theoretical or principled knowledge. Taking the time to do this step well will ensure that your golden thread is more clearly pulled through the earlier sections of your argument, through your data and into your analysis and findings.

 

 

Researching your own ‘backyard’: on bias and ethical dilemmas

Leave a comment

This is a post particularly for those in the social sciences and humanities who may be doing a form of ethnographic research within the context in which they work or study – in other words, doing ‘insider research’ to use Paul Trowler’s term. Researching a context with which one is intimately familiar and in which one has a vested interest can create possible bias and ethical dilemmas which need to be considered by researchers in these situations. The last thing you want, in presenting your completed research, is for your findings to be called into question or invalidated because you have not accounted clearly enough for issues of insider bias, and your own vested interests.

Insider bias and vested interests

In the article cited in this post, Trowler considers issues of bias in data generation. Bias in research can be defined as having only part of the ‘truth’ in your data but treating that part as a whole, ignoring other possibilities or answers because you are prejudiced towards the ones that best represent your interests or investment. If you are working in a context with which you are familiar, especially your own department or faculty, or an organisation in which you have worked or do work, you will have a vested interest in that context. Either you want everyone and everything to look amazing, or perhaps you are unhappy about certain aspects of the ways in which they work and you want your research to show problems and struggles so you have a basis for your unhappiness. Either way, you have to acknowledge going in that you cannot be anything but biased about this research.

bias blindspot

However, acknowledging that you are biased, and detailing what that bias might entail for readers and examiners, does not undermine your position as researcher. By making yourself aware of potential blindspots in your research design – for example the participants you have chosen, or the cases you are including and excluding from your dataset (and why) – you can better head off possible challenges to the validity of your data later on, and you can strengthen your research design choices. Be honest with yourself: there is a balance to strike here between being pragmatic and strategic in choosing research participants, sites, or cases that will be accessible and that will yield the data you need to make your argument, and between choosing too neatly and risking one-sided or myopic data generation. Why these participants, these cases, these sites? Are there others that you know less well that you could include to balance out the familiarity, and increase the validity of your eventual findings? If not, how might you maintain awareness of your ‘insiderness’ and account for this in analysis and discussion later on?

You need to account for these decisions and questions in your methodology, and discuss what it means for your study that you are doing insider research, and that this does imply particular forms of bias. I don’t think you can get away from being biased in these cases, but you can think through how this may affect your data generation processes, and your analysis as well, and share this thinking with your readers frankly and reflexively.

Insider bias and ‘intuitive analysis’

Another point Trowler makes concerns insider ‘intuition’ when analysing the data you have generated and selected for your study. You may be analysing a policy process you were part of, or meetings you sat in on, or projects you were involved in. You have insider knowledge of what was said, the tone of the conversations, background knowledge (and perhaps even gossip) about participants – in other words, you have a kind of cultivated ‘intuition’ about your data set that you reader will not be privy too. Accounting for bias here is crucial, because if you cannot see it, you may rely too much on this insider intuition in analysing your data, and too much of the language of description you are using to convey your theorised findings will be tacit and hidden from the reader. They will then struggle to understand fully on what basis you are claiming that X is an example of poor management, or that Y means that the department is doing well in these particular areas.

ideas

It is thus vital that you get feedback here on whether it is clear to your reader why you are making particular claims, and whether they can see and understand the basis on which you are making such claims. Do they understand your ‘external language of description’ or ‘translation device’ to use Bernstein’s and Maton’s terms respectively? If they do not, you may be relying too much on your insider view of your case or participants, and may need to find a way to step back, and try to see the data you are looking at as more strange and less familiar. Getting help from a supervisor or critical friend who can ask you questions, and expose and critique possible points of bias is a useful way to re-interrogate your data with fresher eyes.

Ethical dilemmas

An ethical dilemma is defined as ‘a choice between two options, both of which will bring a negative result based on society and personal guidelines’. In research, this definition could be nuanced to suggest that an ethical dilemma presents itself when you have to make a decision to protect the interests of your research or the interests of your participants or study site. For example, in an interview with a senior manager you learn information that may be better off staying private and confidential, yet would also add an important and insightful dimension to your findings. What do you do? A participant in your study asks you for help, but to help might be to prejudice that participant’s responses in a later survey or interview, possibly skewing your data. Yet it is your job to help them. Study first, or job first? These are the kinds of dilemmas that can arise when you do research in the same spaces in which you work, and with people you work with and have other responsibilities to outside of your research.

Cheating-clients,-ethical-dilemmas

As researchers we have a duty to be as truthful and ethical in our research as possible. We are working to create and add to knowledge, not to simply maintain the status quo. In your study this may mean being carefully but resolutely critical, reflective and challenging, rather than only saying the palatable or easy things to say. This work is always going to present difficulties and dilemmas, but accounting as far as possible for your own bias and vested interests, and for your own relevant insider knowledge, can create space in your study for the development of your own reflexivity as a researcher, and can bolster rather than undermine the validity and veracity of your findings.

Trowler, P. (2011) Researching your own institution: Higher Education, British Educational Research Association online resource. Available online at [http://www.bera.ac.uk/files/2011/06/researching_your_own_institution_higher_education.pdf]

‘Retrofitting’ your PhD: when you get your data before your theory

2 Comments

I gave a workshop recently to two different groups of students at the same university on building a theoretical framework for a PhD. The two groups of students comprised scholars at very different points in their PhDs, some just starting to think about theory, some sitting with data and trying to get the theory to talk to the data, and others trying to rethink the theory after having analysed their data. One interesting question emerged: what if you have your data before you really have a theoretical framework in place? How do you build a theoretical framework in that case?

I started my PhD with theory, and spent a year working out what my ‘gaze’ was. I believed, and was told, that this was the best way to go about it: to get my gaze and then get my data. In my field, and with my study, this really seemed like the only way to progress. All I had starting out was my own anecdotal issues, problems and questions I wanted answers to, and I needed to try and understand not just what the rest of my field had already done to try and find answers, but what I could do to find my own answers. I needed to have a sense of what kinds of research were possible and what these might entail. I had no idea what data to generate or what to do with it, and could not have started there with my PhD. So I moved from reading the field, to reading the theory, to building an internal language of description, to generating data, to organising and analysing it using the theory to guide me, to reaching conclusions that spoke back to the theory and the field – a closed circle if you will. This seems, to me certainly, the most logical way to do a PhD.

But, I have colleagues and friends who haven’t necessarily followed this path. In their line of work, they have had opportunities to amass small mountains of data: interview transcripts, documents, observation field notes, student essays, exam transcripts and so forth. They have gathered and collected all of these data, and have then tried to find a PhD in the midst of all of it. They are, in other words, trying to ‘retrofit’ a PhD by looking to the data to suggest a question or questions and through these, a path towards a theoryology.

Many people start their doctoral study in my field – education studies – to find answers to very practical or practice-based questions. Like: ‘What kinds of teaching practice would better enable students to learn cumulatively?’ (a version of my own research question) Or: ‘What kinds of feedback practices better enable students to grow as writers in the Sciences?’ And so on. If you are working as a lecturer, facilitator, tutor, writing-respondent, staff advisor or similar, you may have many opportunities to generate or gather data: workshop inputs, feedback questionnaires, your own field notes and reports, student essays and exam submissions, and so on. After a while, you may look at this mountain of data and wonder: ‘Could there be a thesis in all of this? Maybe I need to start thinking about making some order and sense out of all of this’. You may then register for a PhD, searching for and finding a research question in your data, and then begin the process of retrofitting your PhD with substantive theory and a theoryology to help you work back again towards the data so as to tell its story in a coherent way that adds something to your field’s understanding or knowledge of the issues you are concerned with.

The question that emerged in these workshops was: ‘Can you create a theoretical framework if you have worked so far like this, and if so, how?’ I think the answer must be ‘yes’, but the how is the challenging thing. How do you ask your data the right kinds of questions? A good starting point might be to map out your data in some kind of order. Create mind-maps or visual pictures of what data you have and what interests you in that data. Do a basic thematic analysis – what keeps coming up or emerging for you that is a ‘conceptual itch’ or something you really feel you want or need to answer or explore further? Follow this ‘itch’ – can you formulate a question that could be honed into a research question? Once you have a basic research question, you can then move towards reading: what research is being or has been done on this one issue that you have pulled from your data? What methodologies and what theory are the authors doing this research using? What tools have they found helpful? Then, much as you would in a more ‘traditional’ way, you can begin to move from more substantive research and theory towards an ontological or more meta-theoretical level that will enable you to build a holding structure and fit lenses to your theory glasses, such that you have a way of looking at your data and questions that will enable you to see possible answers.

Then you can go back to your data, with a fresh pair of eyes using their theory glasses and re-look at your data, finding perhaps things you expect to see, but also hopefully being surprised and seeing new things that you missed or overlooked before you had the additional dimension or gaze offered by your theoretical or conceptual framing. But working in this ‘retrofitted’ way is potentially tricky: if you have been looking and looking at this data without a firm(ish) theoretically-informed or shaped gaze, can you be surprised by it? Can you approach your research with the curious, tentative ‘I don’t know the answers, but let’s explore this issue to find out’ kind of attitude that a PhD requires? I think, if you do decide to do or are doing a PhD in what I would regard as a middle-to-front sort of way, with data at the middle, then you need to be aware of your own already-established ideas of what is or isn’t ‘real’ or ‘true’, and your own biases informed by your own experience and immersion in your field and your data. You may need to work harder at pulling yourself back, so that you can look at your data afresh, and consider things you may be been blind to, or overlooked before; so that you can create a useful and illuminating conversation between your data and your theory that contributes something to your field.

Retrofitting a PhD is not impossible – there is usually more than one path to take in reaching a goal (especially if you are a social scientist!) – but I would posit that this way has challenges that need to be carefully considered, not least in terms of the extra time the PhD may take, and the additional need to create critical distance from data and ‘findings’ you may already be very attached to.

Data: collecting, gathering or generating?

Leave a comment

I’m thinking about data again – mostly because I am still in the process of collecting/gathering/generating it for my postdoctoral research. I had a conversation with a colleague at a conference I went to recently who talks about ‘generating’ his data – colleagues of mine in my PhD group use this term too – but the default term I use when I am not thinking about it is still ‘collecting’ data. I’m sure this is true for many PhD scholars and even established researchers. I don’t think this is a simple issue of synonyms. I think the term we use can also indicate a stance towards our research, and how we understand our ethical roles as researchers.

Collect (as other PhD bloggers and methods scholars have said) implies a kind of linear, value-free (or at least value-light) approach to data. The data is out there – you just need to go and find it and collect it up. Then you can analyse it and tell your readers what it all means. Collect doesn’t really capture adequately, for me, the ethical dilemmas that can arise, large and small, when you are working in the ‘field’. And one has to ask: is the data just there to be collected up? Does the data pre-exist the study we have framed, the questions we are asking, and the conceptual and analytical lenses we are peering through? I don’t think it does. Scientists in labs don’t just ‘collect’ pre-existing data – experiments often create data. In the social sciences I think the process looks quite different – we don’t have a lab and test tubes etc – but even if we are observing teaching or reading documents, we are not collecting – we are creating. Gathering seems like a less deterministic type of word than collecting, but it has, for me, the same implications. I used this word in my dissertation, and if I could go back I would change it now, having thought some more about all of this.

Generating seems like a better word to use. It implies ‘making’ and ‘creating’ the data – not out of nothing, though; it can carry within it the notions of agency of the researcher as well as the research participants,  and notions of the kinds of values, gazes, lenses, and interests that the parties to the research bring to bear on the process. When we generate data we do so with a particular sense in mind of what we might want to find or see. We have a question we are asking and need to try and answer as fully as possible, and we have already (most of the time) developed a theoretical or conceptual gaze or framework through we we are looking at the data and the study as a whole. We bring particular interests to bear, too. If, as in my study, you are doing research in your own university, with people who are also your colleagues in other parts of your and their working life, there are very particular interests and concerns involved that impact not just on what data you decide to generate, but also how you look at it and write about it later on. You don’t want to offend these colleagues, or uncover issues that might make them look bad or make them uncomfortable. BUT, you also have a responsibility, ethically, to protect not just yourself but also the research you are doing. Uncomfortable data can also be very important data to talk about – it can push and stretch us in our learning and growth even as it discomforts us. But this is not an easy issue, and it has to be thought about carefully when we decide what to look at, how and why.

These kinds of considerations, as one example, definitely influence a researcher’s approach to generating, reading and analysing their data, and it can help to have a term for this part of the research process that captures at least some of the complexity of doing empirical work. For now, I am going to go with others on this and use ‘generating’. Collecting and gathering are too ‘thin’ and capture very little if any of the values, interests, gazes and so forth that researchers and research participants can bring to bear on a study. Making and creating – well, these are synonyms for generating, but at the moment my thinking is that they make it sound too much like we are pulling the data out of nothing, and this is not the case either. The data is not there to be gathered up, nor is it completely absent prior to us doing the research. In generating data, we look at different sources – people, documents, situations – but we bring to bear our own vested interests, values, aims, questions, frameworks and gazes in order to make of what we see something different and hopefully a bit new. We exercise our agency as researchers, not just alone, but in relation to our data as well. Being aware of this, and making this a conscious rather than mechanical or instrumental ‘collection’ process can have a marked impact, for the better I think, on how ethically and responsibly we generate data, analyse it and write about down the line.