Critics Misrepresent Poll Methodology

On Wednesday, the St. Louis Post-Dispatch published an article that called into question the methodology of the poll we commissioned for our recent study about school choice opinions among the Missouri population. From the article:

The critics say the Show-Me Institute’s poll by a research firm, Market Research Insight of Gulf Breeze, Fla., was a "push poll." Push polls phrase questions that steer a survey toward a predetermined, desired outcome.

Jung and others point to the phrasing mixed in to the survey’s 50 questions. They say terms such as "crisis" when discussing public schools have negative connotations.

"You have to wonder about the credibility of a poll like that, in our view," said Brent Ghan of the Missouri School Boards Association.

This first objection is odd. The pollsters didn’t assert that the public schools are in a crisis, or even suggest it. Early in the poll, they asked this question:

Which of the following statements comes closer to representing your personal opinion about public schools in Missouri?

This was followed by a few options: "A Crisis"; "Not a Crisis"; "Critics Exaggerating"; "Doing Very Well"; and "Uncertain". These options were presented in rotating order, a measure intended to help prevent predetermined responses. A poll measuring opinions about school choice policies would be incomplete without gauging respondents’ views on the current state of available schools. In any case, respondents were able to choose any of these options, and the fact that only 26 percent statewide chose "A Crisis" as their response demonstrates that, if this question were somehow a "push" ploy, the people of Missouri weren’t falling for it.

Later in the article someone presents another criticism:

Kenneth Warren, a political science professor at St. Louis University, concluded that the order in which the questions were asked — as they are presented in the "poll details" posted on the Show-Me Institute website — constituted "placement bias."

For example, he said, the survey prefaced one question with a wide range of statistics purporting to demonstrate the economic benefit of school choice. Warren noted that the next question — "Do you think Missouri should or should not have some form of school choice … ?" — was key to supporting the Show-Me Institute’s position on the issue.

"When school choice is presented the way it is in this survey, it becomes a push poll," said Warren.

This seems more plausible — until you realize that these particular questions place at #33 and #34 in a 50-question poll. Almost all of the respondents’ demonstrated support for school choice came much earlier in the poll, beginning with question #4, where 57 percent statewide say they think school choice would work better than a single public school system. This rises into the 60s for parents and minorities. At question #10, 85 percent of respondents statewide indicated they think parents should make the basic decision of which school — or kind of school — that children should attend. This rose to 88 percent for African-American and Hispanic respondents. This is a huge margin of support, early in the poll, without any sort of preparation that could be seen as a  "push."

Not only that, but 11 of the 16 questions that follow the ostensibly objectionable questions, #33 and #34, are entirely demographic in nature. If the poll was meant to "push" people toward desired responses, why would it follow the single question someone hopes to identify as a "push" question with a string of queries entirely unrelated to school choice — questions about age, occupation, income, education level, gender, etc.?

This really seems to be a case of naysayers grasping at straws. They don’t like school choice policies, so they hope to discredit a poll that reveals a strong level of support for school choice. The Post-Dispatch may have fallen for these critics enough to take their tenuous claims seriously in their article, but anybody who takes a substantive look at the actual poll can tell the methodology was sound.

The article also quotes Verne Kennedy, president of Market Research, the firm that conducted the poll, with an astute observation:

"The basic response anyone gives today when they disagree with survey results is to label it a push poll," he said. "That’s the classic response."