Archive for the '28607272' Category

The magic numbers

This [Via wee Phil]

Experts in the math of probability and statistics are well aware of these problems and have for decades expressed concern about them in major journals. Over the years, hundreds of published papers have warned that science’s love affair with statistics has spawned countless illegitimate findings. In fact, if you believe what you read in the scientific literature, you shouldn’t believe what you read in the scientific literature.

I have a very dubious and incomplete knowledge of statistics. Unsurprisingly we had modules of it in medical school. Also unsurprisingly I didn’t go to any of the classes. They were were quite staggeringly dull and I had better things to do like read CS Lewis.

These days I spend a lot of time reading medical studies and trying to critique the numbers and methodologies found there in. There is a tremendous amount of Bad Science out there. And don’t be fooled into believing that it’s just the SCAMs (Supplements, Complementary and Alternative Medicine) – there’s plenty of badly done stuff in the professional and published medical literature.

People say statistics can be used to prove anything. People say that 62% of statistics are made up. These things may be true. I just can’t prove it.

Doctors bow before the throne of the randomised clinical trial. Where a bunch of people get one drug and a bunch of people get placebo and no one knows what anyone got till after the trial is over and the scientists go back and look at how people did on either the drug or the placebo.

If more people lived while on drug then they compare the numbers and calculate the probability that the observed positive results of the drug would be due to chance alone (this involves assuming a null hypothesis) and if this number is less than 5% (ie the chance of the observed results being to chance alone is less than 5%) then it is called a significant difference.

[Apparently this idea was birthed by a chap called Fisher in the 1920s who was looking at agricultural yields with fertiliser]

The problem being that statistical significance has little to do with clinical or actual significance.

If you do a big enough trial involving 1000s of people on a certain drug you can prove a stastistically significant benefit from your drug even if the actual difference between the two groups was 3 people out of 10000 dying in the placebo group and 2 people out of 10000 dying in the wonderdrug group (incidentally you could also quote a 50% relative reduction or a 0.01% absolute reduction in death in this trial).

So you see that the numbers appear very different depending on how you quote them.

The 5% number (p<0.05) is an arbitrary number. Let that sink in. Someone just decided that 5% defined statistical significance. In the name of objectivity and scientific rationalism surely that can’t be right!

But these are the tools we have and the tools we use to misunderstand and spin the medical literature.

This is how lots of people build their CVs and careers – you can’t get anywhere without publishing. And this is how lots of people make large amounts of money from new drugs that are basically stereo-isomers or other barely changed molecules from the previous ones.

Worth reading anyhow.



August 2019
« Sep