How To Distort Polls: The NYT/CBS Poll on Public Sector Workers

The big surprises of yesterday’s NYT/CBS national poll on public sector unions were that 61 percent of adult Americans think that public employee salaries and benefits are “too low” or “about right,” 56 percent oppose cutting public employee salaries and benefits to reduce state deficits, and 40 percent prefer increasing taxes to decreasing spending on other things, among them public employee salaries and benefits. (The poll reports that even a slight majority of Republicans said public employee wages and benefits were “too low” or “just about right.”).

These results raise eyebrows. They seem to contradict the November election results. Perhaps the fickle electorate has changed its mind since November? Last night’s political talk shows were abuzz.

These poll results are so out of line with conventional thinking that they require careful scrutiny. My own (admittedly non-expert) examination raises three red flags:

FIRST: Why are there so many public-employee and union households in the sample?

The NYT/CBS pollsters conducted telephone interviews with 984 respondents using procedures that “in theory, in 19 cases out of 20”will “differ by no more than three percentage points” from an interview of all American adults. Their procedures include adjusting for under- or oversampling of various demographic characteristics. There are no adjustments for public employment, labor union membership, or political identification.

Twenty five percent of the NYT/CBS respondents are from households that had a public employee (versus 17 percent for the U.S.) and twenty percent had a union member in the household (versus 13 percent for the US). If we combine the two (while avoiding double counting as best we can), some 34 percent of the sample are either public employees or union members (versus 23 percent for the US population).

If we assume (rightly or wrongly) that all public employees and union members responded in the most pro-public-employee way to the survey questions, the pro-public-employee majorities wither away without them. The survey therefore shows that Americans who do not have a union member or public employee in the household think public-employee wages and benefits are too high, oppose raising taxes to cover deficits, and favor reduction of collective-bargaining rights.

(To adjust for the oversampling of public employees and union members alone, one should subtract some eleven percentage points from the more favorable public-employee response rates, as I see it).

SECOND: Are Republicans underrepresented?

The NYT/CBS poll asks questions about today’s most politically heated issue. It would therefore seem appropriate to weight the sample to make it 29 percent Republican, 31 percent Democrat, and 38 percent independent (according to recent Gallup reports) before reporting results that are supposed to capture the “national” mood. At a minimum, the NYT/CBS pollsters should provide the breakdowns of their respondents by political affiliations so that users can make their own adjustments if one group is underrepresented. The high public employee and labor union share already suggests an oversampling of Democrats. There is also the fact that Republican response rates to telephone surveys tend to be lower, requiring that their share be adjusted upwards for this reason alone.

Let the NYT/CBS pollsters release these results. I went to NYT.com/polls and failed to find any of these details.

THIRD: Why are results not broken down by political affiliation?

The NTYT/CBS pollsters clearly have results in their hands broken down by political affiliation. They proudly report that “61 percent of those polled — including just over half of Republicans — said they thought the salaries and benefits of most public employees were either “about right” or “too low” for the work they do.” Later they report “there was also strong opposition from independents: 62 percent of them said they opposed taking bargaining rights away from public employee unions.”

If we have the results and respondent shares broken down by political affiliation, we can calculate ourselves the “national” responses adjusted for political affiliation.

The NYT/CBS poll is an example of the not-so-subtle use of polling to support a political cause. This practice is not limited to the left. My suggestion is that next time, NYT/CBS put much more detail on their website (including the cross tabulations that I find sorely missing). Skeptics can therefore examine the results for themselves so that they can draw proper conclusions from the survey.