But can it make your hair grow in three weeks?

Wow. Welcome to the crazy. Today's article is An investigation into the effect of Cupping Therapy as a treatment for Anterior Knee Pain and its potential role in Health Promotion by Ahmed Younis et al. It was published in the estimable Internet Journal of Alternative Medicine, so you know it must be true. At any rate, you know it must be freely accessible online, which is terrific. Such a service to humanity.

If you don't know what cupping is, that's probably because you were born sometime after the dawn of the 20th century. Cupping is... well, let's just ask the authors:

Cupping is an ancient method of treatment that has been used in the treatment and cure of a broad range of conditions; blood diseases such as haemophilia and hypertension, rheumatic conditions ranging from arthritis, sciatica, back pain, migraine, anxiety and general physical and mental well-being. The aim of Cupping is to extract blood that is believed to be harmful from the body which in turn rids the body of potential harm from symptoms leading to a reduction in well-being.

Now, I know what you're thinking^*: they're performing bloodletting? Is skepstat frakking kidding me? I am not frakking kidding you, but the authors are when they claim in their conclusion that:

The efficacy of the treatment of Cupping for Anterior Knee Pain, Range of Movement and well being has been researched and results reveal statistically significant differences in support of Cupping Therapy.

The gist: The authors sucked some blood from 15 participants complaining of knee pain and measured their range of motion and obtained self-reports of pain and well-being before the bloodletting, excuse me, cupping therapy, and three weeks after the bloodletting. I'm sorry, cupping therapy.

The results: Improvement in outcomes across the board! Low p-values! Huzzah!

The problems: As you should expect, there are any number of ginormous problems with the study. We'll cover the two worst:

1. No control group. Let's engage in a thought experiment. Suppose you take 15 random people complaining of acute knee pain (from any source, mind you), and ask them how their knee feels. They'll probably say it hurts. In fact, they already did. Now wave a kosher dill pickle at them and ask them again 3 weeks later. Chances are, they'll say it doesn't hurt quite so much. This could be seen as an example of what statisticians call "regression to the mean," but it's probably more accurately described by what doctors like to call "getting better."
   
   This is why you always need a control group in treatment evaluation. There's absolutely no way to tell how much of the improvement is due to the experimental therapy if there's nothing to compare it to. In some circumstances you might want a control group that receives absolutely no intervention, in some cases a control group that receives placebo (not the same thing as no intervention, by the way), in some cases a control group that receives a different active intervention. But you always need a control group.
   
   In other news, non-controlled studies have now shown that cupping therapy makes the sun set at night and then rise again in the morning.
   
   
2. Missing data. This is a problem that haunts legitimate research as well, but it's particularly bad in this case. The authors recruited 26 potential victims, but only 15 completed the trial. What happened to the other 40% of the sample? Well, 4 of them never showed up (maybe their pain got better?) and 7 didn't show up for follow-up (maybe their pain didn't get better? maybe they became afraid, so afraid of the fake doctors with the real razor blades?). Real scientists would have tried to include the 7 follow-up no-shows in their analyses, often by assuming that the missing subjects had no improvement at all (a so-called last-observation-carried-forward or LOCF analysis, although most statisticians will tell you that LOCF analyses are terrible). Luckily for the bloodletters, this isn't a real journal.
   

Favorite line: Cupping Therapy has no major side effects aside from minimal discomfort due to the method of application of skin cuts to the patient. In cases where the patient's pain threshold is low, a local anaesthetic can be administered.

Least favorite line: Ethical approval was sought from Kings College Research Committee.

You Brits really need to get your ethical act together.


^* Rhetorical device. I am not^** actually using psi powers to discover what you are thinking.

^** Not currently.

Make a statistician cry

My treasured list of statistical putdowns. Use wisely:

1. I have no confidence in your intervals.
   
   
2. Your momma's so fat, her posterior is bimodal.
   
   
3. Your chi-square has no degrees of freedom.
   
   
4. All your priors are improper.
   
   
5. All your posteriors are noninformative.
   
   
6. Nothing you've ever accomplished in your entire life would survive a Bonferroni correction.
   
   
7. All your significant results are one-tailed.
   
   
8. There's no Box-Cox transformation that can fix your kind of non-normality.
   
   
9. I have more power in my pinky than you have in your entire grant proposal.
   
   
10. You're so stupid, you think SPSS is hard.
    
    
11. You're so stupid, you think association implies causality.
    
    
12. You're so stupid, you think you proved the null.
    
    
13. Why don't you take a random walk off a Brownian bridge?
    
    
14. You're non-significant at alpha = .10 (one-tailed).
    

Q: Do you really need informed consent to do research on fake medical education?

In the course of poking around the CAM literature for source material, I found this interesting study in the journal Chiropractic and Osteopathy: Do chiropractic college faculty understand informed consent: a pilot study, by Dana J. Lawrence and Maria A. Hondras.

Short answer to their question: no.

For those who may not know, informed consent is one of the fundamental principles of modern medical ethics, and it's the cornerstone of human subjects research. The need for guidelines requiring informed consent for all human subjects research was first recognized in the Nuremberg Code, in the wake of the horrific medical experiments conducted by the Nazis. The concept has since been refined in subsequent international agreements and in U.S. federal regulations.

When it comes to research on patients, researchers at medical schools are taught (in mandated educational modules everywhere I've heard of) that absolutely any human subjects research has to have prior approval from their Institutional Review Board (IRB). It's the IRB's responsibility to ensure that informed consent is obtained when appropriate, and only the IRB has the power to determine which studies might be exempt from the informed consent requirement. (Exemptions are usually granted in situations where consent in infeasible and risk is minimal, such as anonymous medical record reviews.) In my experience, this is well-understood by all researchers working with patient populations.

The situation becomes murkier to some when the topic is not medical research per se, but medical education research (say, writing down lists of grades on tests to see if medical students are performing better over time). When do you need consent from the medical students? And when do you need IRB approval? In fact, if you want to share the results of any such research, you have to go to the IRB. This is the first rule of research ethics. You don't collect a single piece of research data without clearing it with the IRB first. And the IRB decides whether you need informed consent from your students. (Default position: yes, you do.)

Lawrence and Hondras showed that this principle was very poorly understood at one chiropractic college (Palmer Center for Chiropractic Research in Iowa). Specifically, they found that among faculty survey respondents (and only 55% of faculty responded at all), there was widespread ignorance of policies for medical education research. To cite just a couple of their numbers, 65% of respondents were unsure whether there were any policies in place at all for student consent in education research. Only 27% of respondents correctly noted that students can decline consent for such research!

Unfortunately, I genuinely don't know how much better the situation would be at a real medical school. Points to the back-crackers for taking the time to research this subject.

Move aside, the cream and the clear. Here comes the electroacupuncturist!

I thought I'd start off with something nice and easy for my first post, so I looked up the most recent issue of the Journal of Alternative and Complementary Medicine and picked an article more or less at random: Bilateral Effect of Unilateral Electroacupuncture on Muscle Strength by Li-Ping Huang et al. Performance-enhancing woo!

The gist: The authors randomly assigned 30 young men to two groups. Group 1 received electroacupuncture (that is, acupuncture with a weak electrical current delivered in continuous pulses through the needle) in their right leg three times a week for four weeks. Group 2 received no intervention. Dorsiflexion (foot elevation) strength was measured for each subject, in each leg, before and after the four week period, by means of a homemade device.


The results: Well, how about a mean 21.3% increase in strength in the right legs and a 15.2% increase in strength in the left legs of Group 1, compared to a measly 3.0% right leg and 4.8% left leg increase in Group 2? How's that for impressive, Mr. / Ms. Skeptic? Someone alert Barry Bonds!

The problems: Refreshingly, the statistical analysis was fairly reasonable and appropriate. The authors used a repeated-measures analysis of variance, with timepoint (pre vs. post), leg (right vs. left) and group as factors. There isn't enough detail to fully evaluate their statistics, but the basic idea is correct.

No, the main problem here was blinding. As in, there was no blinding. What does this mean? It means that, at all stages of the experiment, all the participants and all the investigators knew who was getting electroacupuncture and who wasn't. Why is this a problem? Because biases (both conscious and subconscious) on the part of the participants and the investigators can dramatically skew results.

Participants in Group 1 knew they were getting a treatment that was supposed to increase their strength, and they knew that the investigators wanted them to have increased strength at the end of the trial. This is more than enough motivation for most of these participants to try really hard at the post-intervention strength test. Naturally, individual strength performances can be strongly influenced by motivation and willpower. In the extreme, it could even have been enough motivation for some participants to, gasp, exercise in between sessions. Conversely, Group 2 knew that they weren't supposed to get stronger and that the investigators didn't want them to get stronger.

And as for the investigators: they could have been sending subtle (or not so subtle) signals to the participants to try harder (Group 1) or not so hard (Group 2) during the post-intervention strength test. This problem could have been easily mitigated by using blinded, independent assessors for the second strength test. Sadly, this was not the case.

Oh, and one last smallish problem with the study: utter biological implausibility. I mean, come on, you stick needles in the right leg and the left leg gets stronger? Scientist, please.

Favorite line: We did not collect data with respect to the balance of qi, whereas the subjects in this study were apparently healthy.

Inaugural post

Welcome to SkepStat! My plan for this blog is to critique (and perhaps occasionally praise) the statistical methods used in current research articles. Sort of a post-publication statistical peer-review. Unfortunately, too few journals engage in quality pre-publication statistical review, so there should be no shortage of material. I'm going to be focusing on debunking research that seems implausible, poorly conducted, or just silly, especially research into so-called complementary and alternative medicine (CAM). But anything is fair game!

My hope is that this format will provide some really good case studies for introducing or clarifying statistical concepts and methods. So, we'll have some fun, and maybe we'll learn a lesson or two along the way.