A Case for Experimentation in Enrollment Management

In his book, The Victory Lab, author Sasha Issenberg describes a certain consulting industry that thrives on the ambiguity of not knowing what works.

“It allows just about anyone involved to take credit for good results or attribute blame for poor ones, confident of never being proven wrong … Longevity, as well as the aura of wisdom that comes from it, is a political operative’s most valued trait.

“Over a generation, helping Americans choose their leaders has grown into a $6-billion-per-year industry. But the new profession hums along on a mixture of tradition and inertia, unable to learn from its successes or its failures. The tools available to campaign operatives can do little to explain what makes someone vote — and few of the people toiling inside campaign war rooms seem disturbed by this gap in their knowledge.”

He’s describing political consultancy, but does that sound like any other industry you know? How about the enrollment management industry?

In the same way that it’s hard to attribute what impact an individual campaign TV spot or direct mail piece had on voting behavior, it’s just as hard to say if a direct mail piece or email campaign persuaded a student to enroll at a certain university. So we “hum along on a mixture of tradition and inertia.”

The Victory Lab doesn’t stop there. It goes on to describe how political consultancy has reformed in the last 15 years, relying less on gut instincts and more on hard data. The most influential changes have been improvements in predictive modeling as well as the growth of randomized controlled experimentation in enrollment management to test what really works.

The Dying Stopped

Medicine is another industry that hummed along for centuries on a mixture of tradition and inertia. Then in the mid-1800s an obstetrician named Ignaz Semmelweis was horrified to learn when he went to work at a respected hospital that one mother out of 10 was dying of “childbed fever.” He had an audacious idea that would disrupt conventional wisdom.

Ignaz Semmelweis

Ignaz Semmelweis

Author Kurt Vonnegut told his story in an essay for Psychology Today in 1981 titled “A Truly Modern Hero.” The piece was adapted from a commencement address Vonnegut gave to Southampton College earlier that year.

Semmelweis observed hospital routines and began to suspect that doctors were bringing the infection to the patients. He noticed that the doctors often went directly from dissecting corpses in the morgue to examining mothers in the maternity ward. He suggested, as an experiment, that the doctors wash their hands [with disinfectant] before touching the mothers. What could be more insulting?

How dare he make such a suggestion to his social superiors? He was a nobody, you realize.

But all that dying went on and on and Semmelweis, having so far less sense about how to get along with others in this world than you and I would have, kept asking colleagues to wash their hands [with disinfectant]. They at last agreed to do it, in a spirit of lampoonery, of satire, of scorn. How they must have lathered and lathered and scrubbed and scrubbed.

The dying stopped.

Imagine that: the dying stopped. He saved all those lives. Subsequently, it might be said that he has saved millions of lives — including, quite possibly, yours and mine.

Despite that success and the saved lives, experimentation caught on slowly in the world of medicine. Semmelweis was ridiculed in the scientific community for being “unscientific.” Since Louis Pasteur and others had not yet developed the germ theory of disease — also through experimentation — Semmelweis’ findings were only empirical in nature and lacked a theoretical basis. From there it still took 100 years for rules that prescription drugs had to prove their efficacy and safety through experiments.

The Bravery of Controlled Experiments

As for experimentation in enrollment management, we’re still afraid to do true randomized controlled experiments because it means some students won’t receive our message. If we hold out a control group of students that don’t receive a piece of direct mail, it means they have no chance of being influenced by that piece of mail. However, we ignore the opposite risk — that the direct mail we’re spending tens of thousands of dollars on actually has no effect and is just wasted money.

When you compare withholding direct mail from a prospect, it pales in comparison to withholding medicine from a sick person. Yet that’s exactly what our system of medicine does in prescription drug trials.

Before a prescription drug hits the market, it must prove its safety and efficacy through a series of randomized controlled experiments. Some patients receive the drug and a random selection of other patients receives a placebo. (Of course all patients have to consent to the trial and therefore consent to possibly receiving a placebo). If those who receive the drug have significantly better outcomes, the drug can move forward.

How can it ethically be acceptable to withhold medicine from a sick person? It’s because we don’t really know what works otherwise. It would be less ethical to fill the market with snake oil and bloodlettings that offer people false hope instead of progressing toward drugs that actually heal people.

If this sort of experimentation is good enough when people’s lives are on the line or when the future of our country is on the line, it’s good enough when a student’s college enrollment is on the line.

Some of Capture Higher Ed’s core values are bravery, humility, clarity and constant improvement. This touches on all of them. It takes humility to admit we don’t know everything. It takes bravery to question the status quo.

And it’s the clearest way to know if we are actually improving.

By John Foster, Senior Data Analyst, Capture Higher Ed