This is a guest blog post by Andrew Wilson
During the Planning Care hack day, the group I was part of focused on the answers to the “any other comments” questions in the Care Quality Commission’s “Survey of people who use community mental health services 2013.”
Most of the survey questions were yes/no tick boxes or ratings scales and the results are presented showing both the score and where an NHS trust is ranked, relative to all the other trusts nationally.
The “any other comments” responses are freehand text inputs. They address three questions, “Was there anything particularly good about your care?”, “Was their anything that could be improved?” and “Any other comments?”. We had copies of all the comments from 2011 – 2013 in a spreadsheet.
I had several reasons for being interested in the “any other comments” answers. The first is that it’s a way to ask people what they want to say, rather than what the organisers assume people want to talk about.
When I’ve organised very informal feedback conversations with people I’m working with, I always try to finish with “Tell me, what should I have asked?”, and just by writing that now, I realise I should start with, “Tell me what I should ask about?”. Listening to what people want to say seems like a good source of new knowledge and a way to generate innovations.
I’m the sort of person who often fills in the “any other comments” question at the end of feedback forms but then as soon as I send it I often think, “I bet no one reads that”. When I discovered there was a spreadsheet of comments I couldn’t help thinking,“So that’s where all the ‘anything else’ answers go to die.” Although I can report that, when I discreetly asked what happens to them, they do all get read.
The third reason is that working with numerical data, especially using software, takes skill and knowledge. My favourite piece of work from the day was the decision trees produced by Sam Devlin. I really enjoyed hearing about that during the final presentations – it was all new to me and it was a great illustration of what can be done with specialist skills.
I spoke to someone afterwards who had spent the day trying to import the spreadsheets into a another data analysis tool. That’s not much of an outcome for the ticking clock of a hack day, but given a couple of days to get it working, it might have produced unexpected insights that couldn’t have been found any other way.
Without enough of those skills, I was drawn to the human-readable stories in the comments and I think what we ended up doing in our group was looking at the border of human-readable.
We started by “coding” the comments. I sat down with two experts, people who had experience of using mental health services, and we gave each comment a score from 1 to 5 depending on how positive it was.
I’m not a social scientist and our method of coding was completely invalid and made-up, but from memory of a paper I read once, it is possible to use a structured coding method to equalise out personal biases.
After that, the people with experience of the services wrote a comment about what they had read, for example:
“Sense of humour is really important – personal connections matter.”
“Someone who is starting to struggle a bit.”
All of the comments are listen below.
At the same time as we were doing this, Dan May imported all the original comments into some text mining software and produced a word cloud of the most frequently used words.
That was as far as we got on the day but I’d be interested to know about existing work on this kind of thing. I wonder if there is any value in an “explorer” interface for the comments – which is what the word cloud is an example of – and whether reading the comments has been combined with the decision trees in a productive way? Could the experts place the comments at branching points on the decision trees that expressed where things went right, or wrong, and the comments act as a narrative describing what was happening at those points, or why values at the end of the branches were high or low? For example, the kind of thing captured by “The group and social interaction was very important, not just the clinical support?”
Expert comment on the feedback from person with lived experience.
* Being on the receiving end is frustrating. People see the NHS as one big organisation and wonder why professionals can’t communicate with each other. If I’m in quite a poorly and vulnerable state, why am I having to do all the work?
* They don’t want to stop taking medication but they are willing to try other things.
* The group and social interaction was very important, not just the clinical support.
* The government is always talking about mental health being on the same footing as physical health but people don’t actually find this in their daily lives.
* Being denied a helpful form of therapy can be hugely demoralising and frustrating, especially if the justification is cost rather than appropriateness.
* Discharging someone from therapy before the right time can be counterproductive and lead to more help being needed in the future, not less.
* Sense of humour is really important – personal connections matter.