Fake polls are a real problem

Nate Silver

Oct 22, 2026

99

1

7

Share

Still Counting Episode 7 is ready. This episode is a little bit on the longer side. The original plan was to tell the story of the 2026 campaign through the ubiquitous ads that you’re undoubtedly seeing if you’re in a swing state. But then a bunch of polling-related news broke shortly before taping, such as a New York Times/Siena University poll showing strong numbers for Democrats on the generic ballot, and a scandal involving a fake “polling” firm called Wedgewood Polls.

I’ll have some extensive commentary about Wedgewood later in this post. But first, here’s the Still Counting video…

In 2009, I published a story at FiveThirtyEight suggesting that the prolific polling firm Strategic Vision was probably faking its data. It might have been a dumb risk. I never vetted the story with a lawyer, though I had asked Strategic Vision for comment, and their cagey responses reinforced my view that something weird was going on.

I thought the case was highly convincing, however. (I guess back then, I applied more of a “big forgiveness rather than ask permission” mindset.) Strategic Vision “polls” exhibited statistical patterns — namely, being far more likely to contain some trailing digits (such as 5, 7, 8 and 9) than others, including on obscure questions they probably thought would never be scrutinized — that were canonical signs of forged election results in authoritarian regimes. (Human beings are pretty bad at picking truly random numbers.) It was extremely unlikely these results had emerged by chance alone. My “spidey sense” also told me something was a little off about Strategic Vision’s results — though I forget exactly which details led me to that hypothesis.

But I was right — or at least I think that’s a reasonably safe assumption based on their subsequent behavior. The American Association for Public Opinion Research censured Strategic Vision for failing to disclose methodological information, and Strategic Vision never really attempted to provide any convincing refutation. They packed up their tent and left the industry.

The next year, in 2010, “polls” from another prolific firm, Research 2000, were discovered to contain similar anomalies. Research 2000 fought back a little harder, sending me a cease-and-desist letter, but Daily Kos, whom they were “polling” for at the time, sued them before they, too, disappeared. This was actually a bigger scandal than Strategic Vision. Research 2000 had frequently conducted “polls” for local newspapers and news outlets, presumably for a healthy amount of compensation.

Earlier this week, Wedgewood Polls admitted it had fabricated data. It claimed its “polls” were an “experimental project to demonstrate the vulnerabilities, lack of oversight, and overall low barrier to entry in the modern political polling industry.”

My opinion is that Wedgewood’s explanation is bullshit, a pathetic attempt to get ahead of the story after FiftyPlusOne was set to publish a story exposing the fraud. Whether some sort of crime was committed was unclear — Wedgewood allegedly collected crowdfunded money for a poll that may or may not have ever been conducted.1 FiftyPlusOne reports that Wedgewood identified Jack Sims and Andrew Prater as the people behind the firm. Whoever was responsible for fabricating the data probably shouldn't be trusted with anything election- or poll-related again.

These “polls” had made their way into FLIPR, although they weren’t having much influence on the model. When we removed them from our database yesterday, FLIPR’s various averages and forecasts were basically unchanged. That’s a bit confounded because a lot of polls are coming out these days, including a strong series of generic ballot polls for Democrats published before yesterday’s update. Still, FLIPR includes precautions against being overly influenced by any one poll, and new/unknown pollsters start with fairly low ratings by default.

For what it’s worth, here were the Wedgewood “polls” that had entered our database, as compared to the Silver Bulletin averages at the time.

There’s a slight D-leaning “house effect” here, though it’s pretty mild and not particularly consistent. In races where Silver Bulletin was publishing polling averages (there are a couple of caveats here: see the footnote2), their “polls” were about 0.6 points more favorable to the Democratic candidate on average.

One thing that’s a bit unusual is that Wedgewood “polled” several races where there was little or no publicly-available polling at the time. At Silver Bulletin, we turn on the average for a race once there are three distinct polls from at least two distinct polling firms. Four races where Wedgewood issued a “poll” did not meet that distinction. As far as I can tell, in fact, Wedgewood was the first firm of any kind to “poll” Troy Jackson’s race against Susan Collins in Maine as Graham Platner was preparing to drop out; they had Jackson ahead by 5 points.

In Michigan, meanwhile, Wedgewood was relatively early to “poll” Abdul El-Sayed’s race against Mike Rogers, showing him ahead by 5 points, better than the average3 which had AES ahead by 0.4 points instead. A Democratic candidate who dropped out, Mallory McMorrow, has alleged that Wedgewood’s “polls” hurt donor and voter confidence in her campaign.

In the table, I’ve only listed polls that included the eventual general election nominees because I’m not sure how comprehensive our data is on candidates who lost their primaries. Though an early July Wedgewood “poll” did have Francesca Hong leading by 3 points in the general election against Tom Tiffany in the Wisconsin gubernatorial race when other polls at the time showed the race roughly tied.

So there is something of a pattern here of Wedgewood showing better numbers for notably progressive candidates than the consensus, although it’s hard to know how much to make of three races. Not all of their “polls” ran in this direction, however. In Nebraska, they published a September “poll” showing incumbent Republican Gov. Jim Pillen ahead by 17 points against Democrat Lynne Walz, when subsequent polls have shown the race much closer.

Overall, compared with past fake pollsters — which often tried to fit in rather than draw attention to themselves — the Wedgewood “polls” were a bit more narrative-shaping. Though I also wouldn’t overstate this. Before news of the scandal broke on Tuesday, I was vaguely aware of an entity called “Wedgewood Polls” but couldn’t have told you anything about them beyond that. We tend not to dissect individual polls very much at Silver Bulletin4, especially when they come from polling firms we’ve never heard of. And we’d encourage others to take that approach, too. I generally object to focusing too much on any single polling result. Outlier polls tend to receive more attention than polls that align with the consensus. This isn’t necessarily bad because the alternative is too much herding. But there’s no longer any obviously correct, gold-standard way to conduct a poll. “Just plug it into the average” isn’t a perfect heuristic — there are certain polls I find dubious for various reasons. And our averages weight established pollsters with good track records more heavily. But it’s probably the least-worst alternative versus applying some sort of subjective filter to polls, which may reflect an aggregator’s rooting interests or other biases.

How did the Wedgewood data enter our model in the first place?

The literal answer is that we take our polling data from The New York Times, subject occasionally to some manual additions and overrides. This has several advantages. The NYT feed is fast, comprehensive, and staffed by highly skilled professionals. And contrary to what the Wedgewood fraudsters have alleged, the Times does a lot of vetting. (The NYT didn’t include polls from another fake polling firm, Median Strategies, for instance.) In addition, the NYT feed is fairly explicitly modeled on the old FiveThirtyEight database schema and their criteria for poll inclusion, which I helped to design. So it doesn’t make a lot of sense to recreate their efforts when they’re basically already doing the work in basically the same way that we’d do it.

But what are those “manual additions and overrides”? They fall into a couple of categories. We have a list of 11 permanently banned pollsters whom we know or believe the preponderance of evidence suggests fabricated data or engaged in other gross misconduct — our opinion, not necessarily “proof beyond a reasonable doubt”. Most of these polling firms, like Strategic Vision and Research 2000, are no longer active or are rejected by the NYT feed anyway, but FLIPR just ignores them if they enter our database.

On top of that, we have another set of polling firms we consider “nonprofessional.” We aren’t accusing these polls of being fake; we just think they don’t meet the standard of scientific polls. The official Silver Bulletin rule is as follows:

We don’t use DIY polls commissioned by nonprofessional hobbyists on online platforms such as Google Surveys.

As a result, we exclude certain pollsters, like from SoCal Strategies5 and ActiVote, that are sometimes accepted by other polling aggregators. I’m not sure that I’m eager to discuss these choices on a case-by-case basis. If we flag a polling firm as “nonprofessional,” they’ll almost inevitably lobby us and make their case. Sometimes we find this convincing, and sometimes we don’t. On the other hand, we manually add polls from Rasmussen Reports, which aren’t listed by the NYT, owing to a legacy decision by the post-Nate version of FiveThirtyEight that we strongly disagree with.

We will also sometimes add polls if we catch them right before publication of our daily update, and the NYT hasn’t added them yet. But generally, we’re not doing a lot of this. If the NYT hasn’t listed a poll, it may be because it didn’t pass their vetting.6 None of this is that urgent, and we’d rather wait a day or two to see whether the NYT and other aggregators have included the poll lest we add some fake or nonprofessional datapoint.

What Silver Bulletin generally hasn’t been doing, though, is conducting “secondary screening” once a poll passes through the equivalent of the TSA security perimeter — meaning the NYT database. And that’s because the NYT usually does a very good job with its vetting. As the New York Times explained in an editor’s note, when new pollsters enter their database, they’re subject to a fair amount of scrutiny including emails and phone calls.

Of course, someone willing to fake a poll may also be willing to fake answers in response to a vetting process. And this is probably getting easier because of AI (although AI can plausibly also help with fraud detection). Ultimately, thanks to the smart work of FiftyPlusOne, this particular scam was caught, however.

No vetting process is going to bat 1.000, however. What’s wrong with taking an incredibly strict policy? Well, if you were a frequent flier after September 11, you probably know. Vigorous screening can waste time for the vast majority of people doing nothing wrong. And overly strict screening can produce a lot of false positives and contribute to racial or other sorts of profiling.

At the same time, the American aviation system just doesn’t let people board planes without passing through some sort of screening process. It’s not quite “guilty until proven innocent,” but boarding a commercial airline is treated as a privilege and not a right.

Polling is roughly like that too. Overall, within the broader polling/forecasting community, there’s probably more screening and vetting than there was several years ago. New/unknown pollsters do not enter the NYT database without passing through a vetting process, and most of the other aggregators follow a similar if less formal approach7, including Silver Bulletin in cases where we maintain our own original database.8

So the industr