One of the challenges with relying on experts to validate or prioritise problem areas is that this process often leads to contradictory feedback.
Using a stimulus such as a map to guide conversations with experts is of great help in this context because it means that all the individual conversations we have are referring to the same initial set of ideas, making it easier to compare feedback.
Another trick is to be very diligent when it comes to recording feedback. Again, the stimulus can be very helpful here because it makes it easier to record feedback and group it around specific aspects of a problem. This makes it easier to communicate divergent views and reach a consensus about next steps.
To help us record feedback consistently, we ask each interviewee if they are alright with us taking notes or recording the call. If they don’t mind, we create a rough transcript of the interview, with main ideas and quotes highlighted. We do this for every interview.
After we have completed all the intended interviews for that version of the stimulus, we create a summary of the main comments, suggestions, corrections and ideas from all the interviews. We then group these into items that need to be discussed, decided upon, and reflected in the next iteration of the stimulus. We usually end up with two types of items:
- Things (mostly) everyone agrees with: these can be both things that experts are happy with and they think are well defined, or things that they disagree with, but they’re in agreement about how they should change. We then reflect these changes in the next iteration of the stimulus.
- Things (mostly) everyone disagrees with: these are changes to the stimulus/our assumptions where there is disagreement among experts about how to proceed. We then work towards finding a way to reflect this disagreement in the next iteration of the stimulus. We can either share this updated stimulus with the same experts or with new ones to get their feedback on the different options available.
We found that having a transparent record of feedback received and decisions made is incredibly useful as it offers a clear and evidence-based justification for the focus on a particular problem. It can also encourage open conversations around the desired impact of a funding organisation and the extent to which focusing on a particular problem is going to achieve that impact.
