How Confident Are You?
Posted on 12. Feb, 2010 by chiantichiara in Communications
A man I’ve never met before walks into a bar and says to me, “I bet I can guess when your birthday is.” Now I’m not much of a gambler, but bearing in mind he’s only got a 1 in 365 chance of getting it right, I might wager £10 that he can’t do it. However, if that same man walks in and says, “I’m 95% confident I can guess when your birthday is.” I probably wouldn’t bet against him. Being 95% confident in something is pretty much certain, right? Well, not always.
Recently I was asked by my colleagues to deliver a refresher presentation on common statistical methods used when analysing clinical data. Now I should categorically state upfront that I’m not a statistician or any form of mathematical whiz, but out of us all I perhaps have a more natural aptitude for this sort of geekiness. When pulling the slides together I had to delve into the deepest recesses of my mind to try and remember what I learnt at school and university, and by harnessing the power of the internet I was able to fill in the fairly large gaps. What struck me first was just how much I’d forgotten (or perhaps just not retained in the first place), but it also got me thinking about the value of statistics, the degree of trust you can place in them and the terminology surrounding them. In particular, I got a little side tracked on the word ‘confidence’ and the difference in meaning between the common usage and statistical usage. I think it’s fair to say that the difference is quite confusing to the layman. And then what about p-values and statistical significance? Even for the partially informed, the world of medical statistics is a bit of a minefield that needs to be navigated carefully to ensure no misinterpretation occurs.
I’m beginning to believe that a basic grasp of medical statistics should be a requirement of every healthcare PR practitioner. And no, this is not just because I’m doing this presentation! At Aurora, in the last month, we have reviewed, read or referenced 170 clinical papers between us. As we frequently write press releases and backgrounders for the media based on clinical trial data we need to be confident that we understand the data to begin with and that we’re not relaying misinformation. Yes, all materials are reviewed by company medics and regulatory approvers; however I do believe we have a responsibility to understand what P ≤ 0.05 et al. really means.
So does this statistical minefield also exist for doctors and nurses too? As I understand it, statistical confidence intervals provide clinicians with the exact information in a form that helps them decide whether to administer a particular therapy or not. My GP is not only a lovely lady, but a highly trained professional, however how confident am I that she is confident in interpreting confidence intervals? What about calculating relative risk reduction (RRR) vs. absolute risk reduction (ARR) vs. numbers needed to treat (NNT) as is required when applying clinical evidence to the care of individual patients? In layman’s terms, I would say I’m 95% confident in her abilities although I’m not sure whether her statistical skills are called into play every day and if they are therefore not a little rusty. In days of yore, I participated in a meeting with some of the UK’s most senior and respected specialist oncology nurses. As part of the meeting, a clinical statistics refresher lecture was given and this was unanimously voted the most beneficial session of the day. It does raise the question as to whether this should be an area for on-going professional development?
Statistics in general can be easily manipulated to convey as much or as little as you want – 75% of people asked prefer milk chocolate to dark chocolate (but I only asked 4 people) – so they need to be handled with care and a modicum of respect. The more accurately we can interpret them, the clearer the picture will be. Of that I am 100% confident.
Neil Crump
12. Feb, 2010
So my fellow Aurorian. Please tell us the significance of the magic 1 in 20 rule and chance. Take me back to a lecture theatre at the University of Bath circa 1990…
Garry Warburton
03. Mar, 2010
Let me ask you a question in return. How confident do you need to be?
Setting aside the issue of sample sizes, that statistical methods are meant, in any case, to correct for, let’s assume that a realistic power calculation has been made. Then, most of us are pretty OK with the p<0.05. Thus, when we read that (e.g.) a new non-opioid pain killer reduces headaches from 55% in the control group to 25% in the treated group, we do not even look to see what the p value is.
But what if the end-point is something more serious (death is quite serious), and the values are 5.5% in the control group and 2.5% in the treated group? The relative risk is the same but are we as willing to accept a confidence level of 95%, i.e. are we willing to be wrong 1 time out of 20 for a serious event that occurs in about 1 in 20 cases? Or would we feel more comfortable if we had p<0.01 or p<0.001? If your wager was for £1 million, what odds would you accept?
For our treating physicians, it’s even more complicated. They have to consider both benefit and risk. If the non-opioid pain killer above caused gastric bleeding (I can’t spell haemorrhage) in 40% of cases, how does the physicians assessment of the efficacy statistics look now?
This is the problem: on the whole individual physicians treat individual patients (OK, apologies to public/community/occupational health practitioners), and they have to consider benefit and risk, and this in the context of the overall health of the patient. How useful now is the value of p<0.05. Of course it has an indicative value, but statistical results have to be put in the context of the disease, the patient and the (expected) outcome, and our need for “confidence” depends on these factors also.
Both the scientific and communication communities should bear this in mind when preparing their reports.