But I'm back, with a treat. Today we'll be enjoying a woo-woo challenge to clinical trials methodology, in the form of Resonance, Placebo Effects, and Type II Errors: Some Implications from Healing Research for Experimental Methods, published in The Journal of Complementary and Alternative Medicine by William F. Bengston ( an energy healing sociologist) and Margaret Moga (an anatomist and moxibustion enthusiast).
Randomized controlled trials
The randomized controlled trial (RCT) is generally considered to be the gold standard experimental design for medical research. RCTs are widely used and there's a huge literature on RCT methodology (including a journal entirely devoted to the subject), but the basic idea is quite simple.
Let's take a relevant example. Suppose we want to examine the efficacy of, say, laying-on-of-hands "energy healing" for shrinking tumors in mice. Well, efficacy is always efficacy relative to something, so we need to choose an appropriate control treatment. In this example, let's say we're interested in comparing "energy healing" with no treatment at all.
Well, we need to start with a sample of mice with tumors. In animal research, when you want to study animals with a certain disease, you usually start with healthy animals and then give them the disease you want to study. (Generally, this practice would be looked at somewhat unfavorably in human subjects research.) So, we'll take a bunch of mice and inject them all with tumor cells. Stop looking at me like that, animal rights activists. Anyway, maybe one of the experimental mice would have gone on to become the mouse equivalent of Hitler. You never know, do you?
Anyway, the next step is key: we randomly assign the cancer mice to receive either the "energy healing" or no treatment at all. The principle of randomization is deceptively simple, but it has profound implications. The most immediate implication is that, because all personal characteristics of the subjects are randomly distributed between the groups, any observed differences in outcome are likely to be due to the treatment assignment rather than to irrelevant factors like age, sex, socioeconomic status, or whatever the mousy analogues of these might be. It's no exaggeration to say that the introduction of randomization by (principally) R. A. Fisher, was one of the seminal moments in the history of science.
Then we give our treatment group the treatment of interest (i.e., laying-on-of-hands) and our control group the control treatment (i.e., no treatment). After an appropriate length of time, we measure the tumors on all our mice, calculate the remission rate for the treatment group and the control group, and use standard statistical methods (Fisher's exact test, say, although more on this subjet another time) to decide whether the observed difference in remission rates is larger than would be expected due to simple chance differences between the groups.
The inconvenience of negative results
Now suppose you conduct the experiment described above not once, not twice, but four separate times, each time with the same result: many of your "energy healing" treated mice undergo tumor remission, but so do many of your control mice. In fact, in each experiment, the difference between treatment and control groups is smaller than would be expected due to chance ("not statistically significant").
Now, a few possible explanations might spring to mind for these negative results:
- The tumor induction was inadequate in both control and treatment mice to result in a sustained cancer. The majority of mice remitted because their disease was self-limiting. The "energy healing" treatment doesn't work better than no treatment.
- The tumor induction was adequate but some physiological process or trait separate from the treatment but common to all mice resulted in tumor remission in the majority of mice. The "energy healing" treatment doesn't work better than no treatment.
- #1, except the "energy healing" treatment does work better than no treatment, but the effect is small and the sample size was inadequate to determine statistical significance.
- #2, except the "energy healing" treatment does work better than no treatment, but the effect is small and the sample size was inadequate to determine statistical significance.
- The study protocol was broken in some way due to carelessness or fraud.
- THE VERY ACT OF RANDOMIZATION CREATES A MYSTICAL RESONANT BOND BETWEEN THE TREATMENT AND CONTROL MICE WHICH MEANS THAT ANY ENERGY HEALING APPLIED TO THE TREATMENT MICE IS TRANSFERRED TO THE CONTROL MICE. THE TREATMENT WORKS SO WELL THAT IT WORKS EVEN ON MICE THAT DIDN'T ACTUALLY RECEIVE IT!!@$!@@$!!
That's right! Number 6 is obviously gold. The really great thing about the hypothesis underlying number 6 is that, the more negative your results are, the more amazingly effective your treatment must be! The only way things could look better for the treatment is if more control mice remitted. And not only does the hypothesis explain this experiment, it single-handedly accounts for all placebo effects in all clinical trials, ever. This is paradigm-shattering stuff, truly.
But let's not jump to conclusions. We're scientists here. We like the cut of hypothesis 6's jib, sure, but we're not going to just jump in and publish an article proposing an occult relationship between control and treatment groups in RCTs without first making damn sure we can back it up with science. What we need is an experiment....
How do you test an insane hypothesis?
Why, with an insane experiment of course. SILLY! AHAHAHAHAHAAHAHA.
Ok, so here's what we'll do (and, in case you haven't caught up with my rhetorical stylings so far in this post, "what we'll do" translates to "what Bengston and Moga did"). We'll take 30 mice, inject them with tumor cells, and randomly assign half to "energy healing" and half to no treatment. So far so good - it sounds like a replication of our previous experiments.
But wait, you're saying, what about the resonant bond? If our revolutionary hypothesis is correct, the two groups will be mystically connected, all the tumors will remit, and we'll be back to square one. What we need is a TRUE CONTROL GROUP that won't be bonded with the treatment group, resonantly or otherwise. Then we can see if our two original groups are more similar to each other in course of disease than they are to the third control group.
But how to make that third, TRUE control group? Well, why not take another 25 mice and not inject them with cancer cells at all in the first place? Just 25 mice. No tumors. Then we'll follow all three groups, see if remission rates and biological markers are similar in the treatment and bonded control groups and different in the group that, hey, we never gave cancer in the first place. Why this group should be immune to the mystical resonant bond is anyone's guess, of course, but it's worth a shot, right?
Well, ladies and gentlemen, 55 innocent (or, who knows, maybe not so innocent) mice and several inappropriate statistical analyses later (the rest of this is so much fun, I can't even be bothered to critique their statistics), our long hours of work have paid off. Remission rate in the "energy healing" treatment group? 100%. In the bonded control group? 100%, that's all. Just a paltry 100%. And what about the third control group? The group that NEVER HAD CANCER IN THE FIRST PLACE? 0%. Zero. The big zilch. Nada. Not a single one of the mice that NEVER HAD CANCER IN THE FIRST PLACE remitted from their cancer.
Just in case that's not enough to convince you, Bengston and Moga also measured hemoglobin levels and weighed the spleens of a subset of mice from each group at each follow-up. Guess what? The group of tumor-injected mice that had "no" treatment and the group of tumor-injected mice that had "energy healing" treatment? Pretty much the same. And the group that never had tumor cells injected? Different!
An analogy
The setup: In case all of this talk of mice and tumors has gotten a bit esoteric, here's an analogy for what we've just learned. Let's take three cars: car A, car B and car C. We fill the fuel tank of car A with gasoline, plus a fuel additive. We fill the fuel tank of car B with gasoline only. We leave the fuel tank of car C empty.
The experiment: We line the three cars up and race them, to see which can go farthest in 10 minutes. Cars A and B go 20 miles each. Car C doesn't go anywhere.
The conclusion: The fuel additive is effective, and a resonant bond caused Car B to go as far as Car A. The existence of this resonant bond is proved by the fact that Car C didn't go anywhere.
And that just about settles that. Take a bow, Drs. Bengston and Moga. You've really done something here. Really.
Now those of you still reading shall be rewarded with a collection of choice excerpts.
Questioning the logic of experimental design is the last thing we want to do:
This paper does not question the logic of experimental
design. Rather, it suggests that, under some circumstances,
for example, illustrated by placebo effects, the presupposi-
tion of experimental and control group independence is
questionable. It suggests that this violation can occur via the
creation of a “resonant bond” between groups. Resonance,
in turn, can result in a macroscopic entanglement of exper-
imental subjects, so that a stimulus given to one group also
stimulates the other group.
Weeks of practice to master:
As previously reported, the healing-with-intent experi-
mental protocol required that the volunteer healers practice
mental and “directed energy” techniques taught to us by an
experienced healer formerly based in Great Neck, New York.
These techniques did not involve focused visualization, med-
itation, life changes, or belief of any sort. Although they are
straightforward, the mental techniques required weeks of
practice to master and involved a series of routine mental tasks
that were to be practiced simultaneously while placing hands
around the standard plastic mice cages for 1 hour per day.
Our curiosity got the better of us:
Our intent was to keep the control mice
separate for the duration of the experiment and to keep them
particularly hidden from anyone who knew the healing tech-
niques. Our curiosity got the better of us, however, and, within
several weeks of the first experiment, we violated protocol
and visited the control mice. In hindsight, this may have
proved fortuitous, because it inadvertently opened the door
to unexpected phenomena.
Naturally, it's all down to quantum entanglement:
Almost all of the seeming paradoxes of these remissions
disappear if we allow for the possibility of “resonant bond
formation” and “resonant bond dissolution,” which may
serve to entangle or de-entangle subjects. Certainly the no-
tion of “entanglement,” although still quite mysterious, is
widely accepted and hailed for its predictive power on a
quantum level in conventional physics.
You want an explanation of resonant bonds? Well, how about two explanations, smarty-pants?
Consider two possible hy-
potheses: (1) shared experiences among experimental sub-
jects can “bond” them together resonantly; and (2)
consciousness itself, including that of the experimenter, can
delimit the boundaries of experimental subjects, effectively
defining those who are “in” and those who are “out.” Those
who are “in” form something akin to a larger “collective,”
analogous to those formed by colonies of insects, flocks of
birds, and schools of fish.
All your failed studies are belong to us:
Researchers are encouraged to
reexamine their old data within the framework of resonance
to determine whether these phenomena are as extensive as
they now appear to be (e.g., placebos). This reexamination
needs to broaden the question from the difference between
experimental and control subjects to inquire more generally
about the difference between experimental subjects and
“what ought to have happened.”
7 comments:
skepstat has clearly used a resonant bond to make me more skeptical...of this research. How ironic! What a vicious cycle!
What you forget are the historical controls. No mouse ever, in any experiment, lived longer than 27 days with the form of cancer that these mice were injected with. There was documented tumour growth in all the mice in the breast cancer experiments, except that the tumours ulcerated and healed instead of killing the mice. The injection was done by scientists who have been working with this model of cancer for years. In a number of experiments the mice received double the lethal dose. Every single mouse in the treatment groups AND the control groups should have died. Dr. Bengston is now up to 10 experiments at 5 different accredited institutions, including two medical schools, and the results have been replicated in all 10 experiments. Surely you could not accuse ALL the scientists involved of such gross incompetence that they could muck up ten experiments?
Thanks for your comment, anonymous! The reason I give no weight to the "historical controls" you mention is the same reason historical controls are generally not considered in clinical trials: They weren't subjected to exactly the same experimental conditions as the treatment and control animals in the actual study. This is why we run controlled studies, rather than relying on historical controls for research generally.
The specific experimental condition that I'd be most concerned about in this case is just generally being part of a study under the supervision of a credulous researcher with a great deal at stake in convincing others (and, perhaps, himself) that he has magic powers.
I provided a list of possible explanations for the reported results of the experiments in my post, each of which is far more plausible and parsimonious than the existence of so-called "resonant bonds." I would draw your attention to explanations #1 and #5 in particular, as I consider these the most likely. #5 especially, since the article itself documents a brazen willingness to break protocol in other matters (viz., the interaction with control mice who were supposed to be kept in isolation, as quoted toward the bottom of my post).
The scientists were not credulous. In fact, they went out of their way to make sure the mice would die, doubling doses, giving multiple injections, etc. The usual pattern was that after the first experiment succeeded in any one particular place there was an assumption that someone made a mistake, so protocols for setting up the second experiment were more stringently observed. Dr. Bengston had nothing to do with preparing the mice for the experiments. As I said, there have been 10 experiments in 5 different places. You have to make the assumption that all these scientists made the same errors, twice over. You are the statistician, you tell me: what are the odds of that?
One doesn't necessarily have to assume that the entire process and all participants were contaminated with error or fraud. I would certainly rather not believe that. (Although even such an extreme situation would be far more plausible than the conclusion to which the authors jumped.) It would suffice for just one critical component of the process to be critically flawed or just one member of personnel with access to the experimental subjects to have been willing to engage in fraud.
I'm not an expert in tumor induction by any means, but I would put my money on a problem with the cell line used to inject the mice - either at the source, or some systemic error in the handling of the material.
Whatever the cause of the experimental results, the fact remains that the hypothesis of a resonant bond is absurd, and the experiment the authors performed to 'test' that hypothesis was ridiculous. I'm quite familiar with scientists bending over backwards to explain away negative results when they earnestly believe the results should have been other than they are, and it can be painful to watch. But this case was just laugh-out-loud funny.
You are still positing the absurd hypothesis that there was something wrong with all the experiments. Come on, do the math!!!! We are talking 10 experiments over 5 years at 5 different institutions; different cancer lines, and even different cancers. Just do the math. What are the chances of something going wrong with all of them? You are accusing multiple scientists at multiple institutions of incompetence or fraud rather than consider what's staring you in the face: that the researchers may in fact be on to something.
Something clearly happened to the mice: the tumours developed, ulcerated and healed. If there had been no cancer, there would have been no tumours. Tests of selected mice showed viable cancer cells all the way through until full cure.
Something clearly happened to both the treated and the control mice: both developed tumours, and in both the tumours ulcerated and healed.
Hematological and spleen weight comparisons showed, in a statistically significant fashion, that the same process happened in both the treated and the control mice, and that hematological and spleen activity in the uninjected mice differed from the injected groups, both treatment and control,
which would mean that different physiological processes were taking place in the injected and the uninjected mice. The physiological processes taking place in the injected mice were consistent with some kind of immunological activity going on.
Dr. Bengston's oversight in this experiment was that he did not have an off-site control group as he did in previous experiments, although I doubt that that would sway you.
In previous experiments the off-site controls all died on schedule.
You are the statistician; if you are going to pick this apart, pick it apart statistically, not by positing the statistically untenable position that all the scientists involved in these experiments suddenly forgot all their training and screwed up time and time again.
Someone just sent me a VCD of Dr. Bengston's first presentation of the original four experiments in 2001 (before he came up with the resonance theory). It's quite interesting. He says he did the experiments to test the claims of a New York healer who said he could heal cancer and also that his method could be taught to skeptical subjects. Dr. Bengston was the original skeptical subject.
He found mice for the experiment with some difficulty; the cancer model he used was imposed upon him by the lab and the biologist that finally agreed to allow him do the experiments. Then when he found the mice, the healer backed out, afraid of being shown up as a failure, I suppose.
So at that point Bengston had the mice, and no healer. Having no plan B, he decided to sub for the healer himself, since he was the one with the most training in the method. His expectation was that at the most he would be able to slow down the progression of the cancer. Full cure was never expected. And it was an even greater shock to him when some of the controls also remitted. (BTW the healer had told him that he should not be allowed to go anywhere near the controls, but Bengston figured that caution didn't apply to him.) He went to see the controls to find out for himself what dying mice looked like, since his treatment subjects were happily running around the cage with huge ulcers on their side where the tumour had split open.
He says in his talk that he fully believed that the biologists had screwed up. He went to three different labs to do the experiments because he refused to believe his own results. The biologists had not seen anything like it either, and many of them had been working with this cancer model for years.
The VCD has nifty photos of the mice in various stages of cancer and recovery.
BTW Dr. Bengston happens to teach statistics and he calls himself a "failed skeptic". He did the experiments to prove to himself that the stuff didn't work :)
Post a Comment