The results of my PhD scoping workshop on social impact

Some of the data gathered from my workshop

Some of the data gathered from my workshop

As part of my wider Phd research into researcher perspectives on societal impact, I held an initial scoping workshop a couple of weeks ago to further develop my understanding and gather data to inform the writing of my interview questions to be used in later case studies. I found it really useful, so thought I would share with you how and why  I did it. As always, comments very welcome!

Participatory or action research

The idea was to draw upon a participant action research approach, as this approach allows for an emphasis on participant experience and understanding that is integral to my research questions and aims.

This workshop design allows the researcher and participants to work together with two broad aims: to understand the participant’s world, and to learn how to change aspects or carry them out more successfully. In this case, these broad aims were to understand more fully the impact debate and the issues surrounding it, and to learn how to change and improve the understanding, communication and assessment of social impact.

Aims of the workshop

The initial scoping workshop addresses two key sets of aims. Firstly, it directly addresses my research aims as detailed above in the introduction. By holding the workshop and analysing the data gathered I further my own understanding of the impact debate whilst also creating a space in which participants can discuss and explore their impact experiences and opinions.

In addition to this, the workshop addresses some specific aims relating to research design:

• To identify key issues and concerns surrounding social impact for researchers.
• To aid the informed development of a set of questions to be used during the data gathering stage of the case studies.

Structure of the workshop

Although participatory action research is always going to be approached in different ways depending on the context of the subject and participants, there have been attempts to suggest structure templates as a basis to design workshops around. I adapted an eight-step structure (Heron and Reason 2006) into a more compact five-step one to suit the amount of time I had available and to focus more on the participant input than my own research journey. I did however keep a short section in to explain my research aims and to illustrate how the workshop fits into my research journey, as this information is important when ensuring that participants are well informed about how the data gathered will be used.

• Welcome and introductions
• Short presentation on the context of my research and where the workshop fits in as part of my research journey
• Group discussions around what social impact means to the participants and feedback
• Group discussion around which areas of social impact the participants feel need further understanding and feedback
• Summary, next steps for the research and thank you

Data analysis

The majority of the data collected during the workshop was in the form of mind maps and notes written on flipchart paper during group exercises. For analysis of this I have taken direction from a qualitative analysis approach often used for coding different types of data (Bryman 2012). The stages of analysis include the following:

• Transcribe and code at an early stage
• Initial read through of documents, no note taking
• Second read through with marginal notes including key words
• Review the emerging codes

After typing up all the comments from the flipchart sheets, I followed the above stages, resulting in a list of key ideas and questions around each discussion point. I have then summarised these into the following themes:

Exercise 1 – What do we need to consider when
defining social impact?

• University strategy/management
• Impact as part of the research journey
• Metrics/measuring
• Public engagement
• Differences between subjects
• Influencing policy
• Working with industry
• Effects on practice/services
• Data/publication access

Exercise 2 – What are some of the issues and Qs which could
help us understand social impact better?

• University strategy/management
• Impact as part of the research journey
• Metrics/measuring
• Public engagement
• Differences between subjects
• Influencing policy
• Overlap with academic/economic impact
• Defining impact

Many of the issues arising from the two discussions overlapped, indicating that a lot of the key issues surrounding social impact are not seen by the participants as being sufficiently understood.

What next?

I am now using the data gathered from the workshop to write a draft set of semi-structured interview questions to use during my case studies. Overall, it was a really useful experience and has definitely given me more confidence in my research design.

Planning my initial scoping workshop

Exciting times are afoot this week! In just under two weeks time I will be holding my first data gathering exercise in the form of a group workshop to look at opinions around and experiences of social impact from a researcher point of view.

Designing the workshop

There will be around 10-15 participants, including researchers from across the career spectrum including ECRs, Lecturers, Professors and Heads of Research. I chose to sample participants in this way to make sure there would be a diverse range of experience to draw upon.

When designing the workshop I have been drawing on principles of action research, keeping the focus very much on the co-creation of knowledge and exploring areas which the participants believe to be viable and useful. This will ensure that I am exploring areas of importance to researchers, and not just relying on my own assumptions of what the key issues are.

With this in mind, the workshop will focus around 2 key areas:

  • What does social impact mean to you?
  • Which areas need to be more thoroughly understood?

The participants will be discussing these questions in small groups, then feeding back their responses to the whole group. These feedback sessions will be recorded on dictaphones to allow for transcription and analysis later on.

Understanding the research context

One of the things I wanted to make sure of was that the participants would be able to understand why I want to use their opinions and experiences, and how it fits in with my research as a whole. To help with this, I will be using a couple of visual aids:

I developed this ‘impact tree’ as part of a poster presentation for the ARMA conference last year, to help visualise the scope of my research. Hopefully this will help participants to understand what is in and out of scope in terms of defining social impact.

Final impact tree

I will also be using a scanned in copy of my research schedule to show, in a very informal (and owl-covered!) way, how the information gathered from the workshop will feed into my wider research plan.

Research schedule

Hopefully the workshop will be a useful experience for the participants as well as for me – I will certainly do an update post after it is finished.

How many qualitative interviews is enough?

How do you choose the number of participants?

So this week I have mostly been battling with my methodology and research design sections, desperately trying to get a handle on how to justify all the choices I will have to make. You see, probably like many researchers out there, I know what I want to do and how I want to do it – I even know why I want to do it. But….my opinion is not justification for the viva, so I need to wade my way through a huge amount of reading to demonstrate that I have made my decisions wisely, and not just on a whim.

A minefield of qualitative research methods terminology

The reading itself isn’t a problem – a lot of it is really interesting and will definitely help me when I come to actually carrying out my data gathering stages. The issue is trying to remember the differences between all the approaches/methods/designs and understanding that they all overlap each other, not to mention that people use interchangeable terminology for some of them – a sure fire way to confuse someone brand new to sociology!

So I have had to take a bit of a step back this week and take it one question at a time. My current question is – how many interviews/case studies do I want to carry out? How many researchers do I want to work with to gather their experiences? I have been reading a review paper from the National Centre for Research Methods which aims to tackle exactly that: How many qualitative interviews is enough?

 

It all depends on context and your research aims

Something which the paper immediately explains is that, really, there is no clear answer to this, that it depends completely on the aims of your research and the context they sit in. This is something I can definitely relate to. For me, the emphasis is very much on the depth of the data – I want to gather a very rich, deep set of data based on lived experiences – something which I know will be time-consuming in nature. This will natuarlly have an effect on the number of researchers I carry out these activities with.

Predicting the future and the saturation principle

The paper also covers the fact that you wont always be able to predict the number of interviews needed, as this could change during the course of your research. The main reason for this would be if you are adhering to the saturation principle; in that you continue to carry out interviews until you stop receiving any new or different information. In my case, I don’t think this would be an option as I am gathering lived experience data and, as far as I can guess, there would always be new experiences and difference emerging, so no cut off point would be reached.

Quality over quantity

There does not seem to be a straightforward answer to my questions – I think for now, the answer is to focus on the fact that I am gathering ‘how’ and ‘why’ data, rather than ‘what’ data, so the depth of the data itself is more significant than the numbers. One way I might choose to frame the number of studies is to consider subject areas/schools/departments, as I am keen to frame these lived experiences in the context of the academic field to ensure the outcomes are relevant to academics and practitioners in their own fields.

 

Method in the madness?

So today I had a very useful (as always) meeting with my mentor, and we used most of the time to discuss issues surrounding the fact that my PhD topic sits within the social sciences, and I don’t have a social sciences background. Whilst this isn’t a problem in itself, it does present some extra challenges around getting to grips with methodologies and research design.

Due to the focus of my research being on an approach tailored to different subject areas, I have naturally inclined towards a case study approach to ensure my work is both tailored and tried and tested. My mentor explained that there are several different routes to take with a case study approach, and that I will need to understand and back up my choice to make sure I can justify it in the long run.

Up to now, because I felt my method was fairly straight forward, my focus has been solely on subject area research, rather than any research around methodology and design. I think in order to feel fully prepared for January, when this begins in earnest, I need to split my time into two streams and make sure I am covering both how I am going to carry out my work, and what my work will be. Hopefully, doing some reading around the theory behind different methodologies will allow me to back up my decision on a case study methodology.

Christine Meyer talks about how case study methodology can be seen as inherently flawed:

“… there are virtually no specific requirements guiding case research. This is both the strength & the weakness of this approach. It is a strength because it allows tailoring the design & data collection procedures to the research questions. On the other hand, this approach has resulted in many poor case studies, leaving it open to criticism, especially from the quantitative field of research. ”

My immediate feelings around this are that it is most likely still the right choice for me, I just need to be rock solid on why it is the right choice.

If anyone reading this has experience of using a case study approach and would like to share their opinions, that would be great.