Category: data analysis

Developing well-constructed data gathering tools, or methods, for your study

1 Comment

I spent the better part of last week working with emerging researchers who are at the stage of their PhD work where they are either working out what data they will need and how to get it, or sitting with all their data and working out how to make sense of it. So, we are talking theory, literature, methodology, analysis, meaning making, and also planning. In this post I want to focus on planning your data gathering phase, specifically developing ‘instruments’, such as questionnaires, interview schedules and so on.

tools

Whether your proposed study is quantitative, qualitative or mixed methods,  you will need some kind of data to base your thesis argument on. Examples may include data gathered from documents in the media, in archives, or from official sources; interviews and/or focus groups; statistical datasets; or surveys. Whatever data your research question tells you to generate, so as to find an answer, you need to think very carefully about how your theory and literature can be drawn into developing the instruments you will use to generate or gather this data.

In a lot of the postgraduate writing I have read and given feedback on, there are two main trends I have noticed in the development of research methods. The first is what I considered ‘too much theory’, and the other ‘not quite enough’. In the first instance, this is seen in researchers putting technical or conceptual terms into their interview questions, and actually asking the research questions in the survey form or interview schedule. For example: ‘Do you think that X political party believes in principle of non-racialism?’ Firstly, this was the overall research question, more or less. Secondly, this researcher wanted to interview students on campus, and needed to seriously think about whether this question would yield any useful data  – would her participants know what she meant by ‘the principle of non-racialism’ as she understood it theoretically, or even have the relevant contextual knowledge? Let’s unpack this a bit, before moving on to trend #2.

The first issue here is that you are not a reporter, you are a researcher. This means you are theorising and abstracting from your data to find an answer that has significance beyond your case study or examples. Your research questions are thus developed out of a deep engagement with relevant research and theory in your field that enables you to see both the ‘bigger picture’ as well as your specific piece of it. If you ask people to answer your research question, without a shared understanding of the technical/conceptual/theoretical terms and their meanings, you may well end up conflating their versions of these with your own, reporting on what they say as being a kind of ‘truth’, rather than trying to elicit, through theorising, valid, robust and substantiated answers to your research questions, using their input.

This connects to the second issue: it is your job to answer your research question, and it is your participants’ job to tell you what they know about relevant or related issues that reference your research question. For example, if you want to know what kinds of knowledge need to be part of an inclusive curriculum, you don’t ask this exact question in interviews with lecturers. Rather, you need to try and find out the answer by asking them to share their curriculum design process with you, talk you through how they decide what to include and exclude, ask them about their views on student learning, and university culture, and the role of the curriculum, and knowledge, in education. This rich data will give you far more with which to find an answer to that question than asking it right out could. You ask around your research questions, using theory and literature to help you devise sensible, accessible and research-relevant questions. This also goes for criteria for selecting and collating documents to research, should you be doing a study that does not involve people directly.

analysis of data

Photo by Startup Stock Photos from Pexels

The second trend is ‘not enough theory’. This tends to take the form of having theory that indicates a certain approach to generating data, yet not using or evidencing this theory in your research instruments.  For example structuralist theories would require you finding out what kinds of structures lie beneath the surface of everyday life and events, and also perhaps how they shape people, events and so on. An example of disconnected interview questions could be asking people whether they enjoy working in their university, and whether there are any issues they feel could be addressed and why, and what their ideal job conditions would be, etc., rather than using the theoretical insights to focus, for example, on how they experience doing research and teaching, and what kinds of support they get from their department, and what kinds of support they feel they need and where that does and should come from, etc. You need to come back to using the theory to make sense of your data, through analysis, so you need to ensure that you use the theory to help you create clear, unambiguous, focused questions that will get your participants, or documents, talking to you about what matters to your study. Disconnecting the research instruments from your theory, and from the point of the research, may lead to a frustrating analysis process where the data will be too ‘thin’ or off point to really enable a rich analysis.

Data gathering tools, or methods for getting the data you need to answer your research questions, is a crucial part of a postgraduate research study. Our data gives us a slice of the bigger research body we are connecting our study to, and enables us to say something about a larger phenomenon or set of meanings that can push collective knowledge forward, or challenge existing knowledge. This is where we make a significant part of our overall contribution to knowledge, so it is really important to see these instruments, or methods, not as technical or arbitrary requirements for some ethics committee. Rather, we need to conceptualise them as tools for putting our methodology into action, informed and guided by both the literature our study is situated within as well as what counts as our theoretical or principled knowledge. Taking the time to do this step well will ensure that your golden thread is more clearly pulled through the earlier sections of your argument, through your data and into your analysis and findings.

 

 

Fairy castles, ramshackle cottages and writing in the real world

Leave a comment

I have this problem: I am not always a huge fan of reality. It’s often far less interesting and well-ordered than the world I can create in my head. For example, it can take 2 years to publish a paper – writing, revising, reviews, more (crushing at times) feedback, more revising, and this goes on and on, paper after paper. It’s tiring, and hard. But, in my head, I write a paper that is erudite and important, and journal editors and reviewers like it a lot, and it gets published within a year, and cited a lot. This sounds silly, right? It is, kind of.

I wrote ages ago about PhD fantasies, and why you should have them. In that post I argue that, while they can be distracting and even perhaps paralysing if you indulge in them too often, and for too long, fantasies can be useful motivation tools. Imagining the eventual future, as a place where we are successful, and have achieved something we are currently struggling with can push us towards that goal.  I think we need fantasy – what I call my ‘fairy castles’ – because fantasies are, at their core, creative acts. I could imagine a fairy castle of writing that goes well, is invigorating and stimulating, and that makes me feel clever, accomplished and productive. In reality, right now, I’m more like in a ramshackle cottage in the woods just trying to get the fire going with damp wood. (I have also been watching too much Once Upon a Time). The reality and the fantasy in my writing life do not align often enough.

fairy castle 2

But, they do align. And I believe that my fairy castle papers -and Book Manuscript, and the New Project I Will Start Planning – are a vital part of creating a reality that will actually be productive, and result in written work that will eventually be published.

The fairy castle and the ramshackle cottage can be part of the same ‘writing realm’ that all writers inhabit. There are always good writing days, where you can concentrate, and make sense, and feel like progress is really being made with the piece you are working on. And there are bad days, where none of the words see  to come out right, or at all. I am starting to really get that these two kinds of days go together – they have to. It can’t all be fairy castles and magical days of erudite brilliance, but by the same token, it can’t all be days of smoky damp fires and frustration. You have to learn from the bad days, to make the good days more frequent, and useful.

The work, to follow the metaphor, is to bring the cottage closer to the castle; to bring it into the walls of the city, like one of those small houses in the shadow of the queen’s castle in medieval time.

fairy castle

I have a plan I am going to try to follow to start inching my writing fantasies and writing reality closer to one another. Firstly, I am going to write down the fantasy in my research journal: finished book chapter by February, and a book proposal and draft chapter by March. Let’s start small, so we don’t crush the whole enterprise at the outset. I am going to outline the steps I need to take to actually get there – how many words, what do I have, what do I need to read, write, do. Then, I am going to stop thinking about it all, and spend an hour a day – two pomodoros – writing. It will be awful at first. I will feel stuck, and frustrated, and cross with myself. The ramshackle cottage will feel like it is falling apart. But, if I keep slogging away at this, the cottage walls will get a bit stronger, maybe the fire will get going at last, and the fantasy of the finished work will start to become more attainable. (And I will probably feel much better about indulging in my binges of Once Upon A Time if I use them as rewards for actually writing, rather than as ways to escape writing!)

The cottage will always be a cottage – it will never turn into the fairy castle up on the hill. I am not sure it should. I think we need the struggles and frustrations to push us across important thresholds in our learning and thinking – about theory, methodology, the nature of our research, the process of actually writing, and so on. The struggles do make the victories that much sweeter, I must say, and they help me to appreciate that this is a real job. Being an active writer and researcher is work, hard work most days, and it is valuable work. To me, and hopefully to others in my field too.

So, my plan for this year is both simple, and really hard: to strive to create a writing realm for myself where the cottage I really live in is closer to the castle I admire, and often wish I lived in, so that they co-exist in a mutually beneficial space, where the fantasy feeds the reality, rather than keeping me from it. Or, where the words become sentences, the sentences become paragraphs, and the paragraphs become my published work.

 

Putting your theory to work in analysis

1 Comment

You now have generated data – in some form, whether primary or secondary – and now you need to code and make sense of it; you need to put it to the task of answering your research question(s). In other words: analysis. This was the toughest part of my own PhD: I had a mountain of data – how to choose the right pieces? What to say about them? How to make sense of them in relation to my research questions?

This is where theory and concepts come into their own in a PhD or MA. You will have some form of theoretical or conceptual framework (for clarity on theory and concepts, how they differ and work together, please watch this short video). Where students often go off track, though, is not using these concepts or theory to do the work in analysis. The theoretical or conceptual framework ends up standing alone, and some form of thematic description of the data is made, with a rather thin version of analysis. In this situation, it may be difficult to offer a credible answer to your research question.

Analysis is, in essence, an act of sense-making. It requires you to move beyond a common sense, everyday understanding of the world, and your data – the level of the descriptive – to a theorised, non-common sense understanding – the level of the analytical (and critical). Analysis means connecting the specific (your study and its data) with the general (a phenomenon, theory, concept, way of looking at the world) that can help to explain how the specific fits in with, or challenges, or exemplifies the general. If you do not make this move, all you may end up with is a set of data that describe a tiny piece of the world, but with little or no relevance to anyone else’s research except perhaps the few other people researching the same thing you are.

theory specs 2

So, how might you ‘do’ analysis?

Imagine you are doing a study on the role of reflective learning in building students’ capacity to critique and create professional knowledge that encourages ongoing learning and problem-solving. ‘Reflection’, or ‘reflective practice’ would be a key concept, as would ‘professional knowledge’, ‘problem-solving’, and ‘learning’. These have generalised, or conceptual meanings that could apply in a range of ways, depending of the parameters and questions of a specific study. Thus, they can do analytical work, helping you to theorise as you answer your research questions.

Then imagine your data set is assessment tasks completed by students in social work and accounting, as two professional disciplines which require adaptive, ongoing learning and problem-solving. You now need a way of employing your key concepts in analysis. You could look at the intentions of the task questions – how they do, or do not, explicitly or actively enable or encourage problem-solving and reflective thinking and learning, and then look at students’ responses and see the extent to which the desired forms of learning are visible or not. This would yield useful findings to feed back to these disciplines in using assessment more effectively.

To reach theorised findings that go beyond describing what the tasks and the student writing said, and conjecture about what the tasks and written responses mean in relation to your study’s understanding of professional knowledge, learning, problem-solving and reflection, you need to start with questions.

theory giphy

For example: these tasks seem to be using direction words such as ‘name’, ‘list’, ‘describe’, ‘mention’, which require mainly memorising, or learning the notes in a rote manner. What kind of learning would this encourage? What impact would this have on students’ ability to move on to more analytical tasks? Is there a progression from ‘memorisation’ towards ‘problem-solving’ or using knowledge to reflect on and learn from case studies etc? What kind of progression is there? Is it sensible, or not, and how could this affect students learning? And so on.

You could then present the data: e.g., this is the task, and this is when students work on this task in the semester or progression of the course, and this is the task that follows (show us what these look like by copying them out, or including photographs). This part of the analysis is quite descriptive. But then you pose and answer relevant questions guided by your overall research objectives: if these two disciplines – social work and accounting – require professional learning and knowledge that is built through reflection, and the capacity to USE rather than just KNOW the knowledge in the field so that professional can adapt, continue learning, and solve complex problems, what kinds of assessment tasks are needed in higher education? Do the tasks students are doing in the courses I am studying here do these kinds of tasks? If yes, how are they working to build the rights kinds of knowledge, skills and aptitudes? If no, what might be the outcome for these students when they graduate and move into the professions? You then have to use the concepts you have pulled together to create a theorised understanding of professional reflective learning to pose credible answers, that are substantiated with your data (as evidence). This is the act of analysis.

analysis

In both qualitative and quantitative studies, the theory or concepts you choose, and the data you generate, are informed by your research aims and objectives. And in both kinds of studies, analysis requires moving beyond description to say something useful about what your data means in relation to the general phenomenon you are connecting with, and that informs your theorisation (student learning, climate change, democratic governance, etc). Thus, you need to work – iteratively and in incremental stages – to bring your theory to your data, to make sense of the data in relation to the theory so that your study can make a contribution that speaks both to those within your research space, and those beyond it who can draw useful conclusions and lessons even if their data come from somewhere else.

 

Paper writing IV: analysing data

1 Comment

One of the trickiest areas for researchers working with data – either primary or secondary (data you have generated in ‘the field’, or that gleaned from texts etc) – is the analysis of that data. It can be a significant challenge to move from redescribing findings, observations or results, to showing the reader what these mean in the context of the argument that is being made, and the field into which the research fits. There are a few moves that need to be made in constructing an analysis, and these will be unpacked in this post.

Often, in empirical research, we make our contribution to knowledge in our field through the data we generate, and analyse. Especially in the social sciences, we take well-known theories and established methodologies and use these to look at new cases – adding incrementally to the body of knowledge in our field. Thus, analysis is a really important thing to get right: if all we do is describe our data, without indicating how it adds to knowledge in useful ways, what kind of contribution will we be making? How will our research really benefit peers and fellow researchers? After all, we don’t write papers just to get published. We conduct research and publish it so that our work can influence and shape the work of others, even in small ways. We write and publish to join a productive conversation about the research we are doing, and to connect our research with other research, and knowledge.

data 1

How to make a contribution to knowledge that really counts, though?

First things first, you can’t use all your data in one paper (or even in one thesis). You will need to choose the most relevant data and use it to further illustrate and consolidate your argument. But how do you make this choice – what data should you use, and why? The key tool used to make all the choices in a paper (or thesis) – from relevant literature, to methodology and methods, to data for analysis – is the argument you are making. You need to have, in one or two sentences, a very clear argument (sometimes referred to as a problem statement, or a main claim). In essence, whatever you call it, this is the central point of your paper. To make this point, succinctly and persuasively, you need to craft, section by section, support for this argument, so that you reader believes it to be valid and worth engaging with.

So, you have worked out your argument in succinct form, and have chosen relevant section of data that you feel best make or illustrate that argument. Now what? In the analysis section, you are making your data mean something quite specific: you are not just telling us what the data says (we can probably work that out from reading the quotes or excerpts you are including in the paper). To make meaning through analysis, you need to connect the specific with the general. By this I mean that your data is specific – to your research problem and your consequent choice of case study, or experiment, or archival search and so on. It tells us something about a small slice of the world. But, if all we did in our papers was describe small slices of the world, we would all be doing rather isolated or disconnected research. This would defeat the aim of research to build knowledge, and forge connections between fields, countries, studies and so on. Thus, we have to use our specific data to speak back to a more general or broader phenomenon or conversation.

data 2

The best, and most accepted way, of making meaning of your data is through theorising. To begin theorising your data, you need to start by asking yourself: What does this data mean? Are these meanings valid, and why? There are different kinds of theory, of course, and too many to go into here, but the main thing to consider in ‘theorising’ your data is that you need a point of reference against which to critically think about and discuss your data: you need to be able to connect the specifics of your data with a relevant general phenomenon, explanation, frame of reference, etc. You don’t necessarily need a big theory, like constructivism or social realism; you could simply have a few connected concepts, like ‘reflection’, ‘learning’ and ‘practice’ for example; but you do need a way of lifting your discussion out of the common sense, descriptive realm into the critical, analytical realm that shows that reader why and how the data support your argument, and add knowledge to your field.

Analysis and theorising data is an iterative process, whether you are working qualitatively or quantitatively. It can be difficult, confusing, and take time. This is par for the course: a strong, well-supported analysis should take time. Don’t worry if you can’t make the chosen data make sense in the first go: you may well need to read, and re-read your data, and write several drafts of this section of the paper (preferably with critical feedback) before you can be confident of your analysis. But don’t settle for the quick-fix, thin analysis that draft one might produce. Keep at it, and strive for a stronger, more influential contribution to your field. In the long run, it’ll be worth more to you,to your peers, and to your field.

Researching your own ‘backyard’: on bias and ethical dilemmas

Leave a comment

This is a post particularly for those in the social sciences and humanities who may be doing a form of ethnographic research within the context in which they work or study – in other words, doing ‘insider research’ to use Paul Trowler’s term. Researching a context with which one is intimately familiar and in which one has a vested interest can create possible bias and ethical dilemmas which need to be considered by researchers in these situations. The last thing you want, in presenting your completed research, is for your findings to be called into question or invalidated because you have not accounted clearly enough for issues of insider bias, and your own vested interests.

Insider bias and vested interests

In the article cited in this post, Trowler considers issues of bias in data generation. Bias in research can be defined as having only part of the ‘truth’ in your data but treating that part as a whole, ignoring other possibilities or answers because you are prejudiced towards the ones that best represent your interests or investment. If you are working in a context with which you are familiar, especially your own department or faculty, or an organisation in which you have worked or do work, you will have a vested interest in that context. Either you want everyone and everything to look amazing, or perhaps you are unhappy about certain aspects of the ways in which they work and you want your research to show problems and struggles so you have a basis for your unhappiness. Either way, you have to acknowledge going in that you cannot be anything but biased about this research.

bias blindspot

However, acknowledging that you are biased, and detailing what that bias might entail for readers and examiners, does not undermine your position as researcher. By making yourself aware of potential blindspots in your research design – for example the participants you have chosen, or the cases you are including and excluding from your dataset (and why) – you can better head off possible challenges to the validity of your data later on, and you can strengthen your research design choices. Be honest with yourself: there is a balance to strike here between being pragmatic and strategic in choosing research participants, sites, or cases that will be accessible and that will yield the data you need to make your argument, and between choosing too neatly and risking one-sided or myopic data generation. Why these participants, these cases, these sites? Are there others that you know less well that you could include to balance out the familiarity, and increase the validity of your eventual findings? If not, how might you maintain awareness of your ‘insiderness’ and account for this in analysis and discussion later on?

You need to account for these decisions and questions in your methodology, and discuss what it means for your study that you are doing insider research, and that this does imply particular forms of bias. I don’t think you can get away from being biased in these cases, but you can think through how this may affect your data generation processes, and your analysis as well, and share this thinking with your readers frankly and reflexively.

Insider bias and ‘intuitive analysis’

Another point Trowler makes concerns insider ‘intuition’ when analysing the data you have generated and selected for your study. You may be analysing a policy process you were part of, or meetings you sat in on, or projects you were involved in. You have insider knowledge of what was said, the tone of the conversations, background knowledge (and perhaps even gossip) about participants – in other words, you have a kind of cultivated ‘intuition’ about your data set that you reader will not be privy too. Accounting for bias here is crucial, because if you cannot see it, you may rely too much on this insider intuition in analysing your data, and too much of the language of description you are using to convey your theorised findings will be tacit and hidden from the reader. They will then struggle to understand fully on what basis you are claiming that X is an example of poor management, or that Y means that the department is doing well in these particular areas.

ideas

It is thus vital that you get feedback here on whether it is clear to your reader why you are making particular claims, and whether they can see and understand the basis on which you are making such claims. Do they understand your ‘external language of description’ or ‘translation device’ to use Bernstein’s and Maton’s terms respectively? If they do not, you may be relying too much on your insider view of your case or participants, and may need to find a way to step back, and try to see the data you are looking at as more strange and less familiar. Getting help from a supervisor or critical friend who can ask you questions, and expose and critique possible points of bias is a useful way to re-interrogate your data with fresher eyes.

Ethical dilemmas

An ethical dilemma is defined as ‘a choice between two options, both of which will bring a negative result based on society and personal guidelines’. In research, this definition could be nuanced to suggest that an ethical dilemma presents itself when you have to make a decision to protect the interests of your research or the interests of your participants or study site. For example, in an interview with a senior manager you learn information that may be better off staying private and confidential, yet would also add an important and insightful dimension to your findings. What do you do? A participant in your study asks you for help, but to help might be to prejudice that participant’s responses in a later survey or interview, possibly skewing your data. Yet it is your job to help them. Study first, or job first? These are the kinds of dilemmas that can arise when you do research in the same spaces in which you work, and with people you work with and have other responsibilities to outside of your research.

Cheating-clients,-ethical-dilemmas

As researchers we have a duty to be as truthful and ethical in our research as possible. We are working to create and add to knowledge, not to simply maintain the status quo. In your study this may mean being carefully but resolutely critical, reflective and challenging, rather than only saying the palatable or easy things to say. This work is always going to present difficulties and dilemmas, but accounting as far as possible for your own bias and vested interests, and for your own relevant insider knowledge, can create space in your study for the development of your own reflexivity as a researcher, and can bolster rather than undermine the validity and veracity of your findings.

Trowler, P. (2011) Researching your own institution: Higher Education, British Educational Research Association online resource. Available online at [http://www.bera.ac.uk/files/2011/06/researching_your_own_institution_higher_education.pdf]