Updated: Suspicion over McKinsey methodology causes stir

It’s rare that anything related to the benefits industry becomes embroiled in a scandal. It’s also uncommon for benefits-related research to be called into question.

However, today, I offer you two common clichés:
1. There’s a first time for everything.
2. There are lies, damn lies and statistics.

Here’s the scoop. Last week, consulting firm McKinsey released the findings of a report that revealed some 30% of employers planned to drop their health benefits once health insurance exchanges created under the Patient Protection and Affordable Care Act open in 2014.

The scandal isn’t in the fact that McKinsey’s findings run counter to other surveys, which have shown employers will either take a wait-and-see approach or maintain coverage. Rather, the watercooler chatter comes from suspicions that the firm’s data collecting may be flawed at best and downright shoddy at worst.

According to reporting from D.C.-based online outlet Talking Points Memo, the McKinsey survey on employers’ post-exchange intentions “wasn't designed in away that would allow it to be peer review published or cited academically,” and that it “was devised by consultants without particular expertise in this area, not by the firm's health experts.” In addition, “the firm may have reached its outlying conclusion by basing its questions on the firm's own advice to clients on how best to arbitrage the new reforms.”

This, from internal sources at McKinsey itself, TPM reports.

Obviously, rather than let an anonymous source set the record, McKinsey released the survey materials and methodology to clear things up, right? Wrong. A McKinsey spokesperson told TPM that the survey materials are proprietary and thus won’t be released and will allow the poll to “speak for itself.”

 This whole business is a good reminder that controversy isn’t limited to celebrities and members of Congress, and that statistics can be slippery.

What do you think? Were you wary of the McKinsey data from the start? Does the controversy change your mind at all about the survey’s validity? Or, does the whole situation not amount to a hill of beans? Share your thoughts in the comments.

UPDATE: After repeatedly refusing to comment on the controversy surrounding the survey, McKinsey issued a statement on June 20 defending its methodology and saying comparing the results to other research on the topic is unfair. 

"Comparing the McKinsey survey to economic estimates, such as the CBO's, is comparing apples to oranges," McKinsey said. "We understand how the language in the article could lead the reader to think the research was a prediction, but it is not."

Click here to read the full story on McKinsey's defense of the survey.

For reprint and licensing requests for this article, click here.