Does Reddit have an astroturfing problem? What the data suggests

One chef's-knife brand gets 31% of its "what should I buy" mentions from 5% of the accounts, four times what chance predicts. So I pulled those accounts' full Reddit histories.

This article was drafted with AI from my outline and the run logs, then edited. If that bothers you, stop here. The numbers come from the linked repositories, and every one of them can be recomputed.

I like to cook, and cooking turned into an obsession with high-end Japanese chef's knives. When I want to buy a knife, or anything else, I type the product name into Google and add the word "reddit". A lot of people do this. Mike Riggs wrote it up in Reason in 2022 as "the Reddit hack", after Dmitri Brereton's essay on Google search made the same point: a query with "reddit" on the end returns humans instead of affiliate pages. The bet behind the habit is that a stranger in r/chefknives has no reason to lie to you about a knife.

That bet has an obvious weak point. If Reddit is where buyers go for unpaid opinions, Reddit is where a brand would want to plant paid ones. I wanted to know whether the knife subreddits I read, and scrape for New Knife Day, show any sign of that. Not a hunch about one suspicious comment, but something I could count and someone else could recount.

Last year I fine-tuned a small named-entity model, GLiNER, to pull brands, models and steels out of knife comments. From "picked up a Mazaki in white #2, way better than my old Fibrox" it returns Mazaki as a brand, Fibrox as a model and white #2 as a steel. That model runs over every comment the New Knife Day scraper collects from six subreddits: r/knives, r/knifeclub, r/chefknives, r/japaneseknives, r/FixedBladeEdc and r/KnifeSteels. The write-up on that model is here.

So for every comment I already have who wrote it, which brands it names, and whether the thread it sits in is someone asking what to buy. That is enough to ask a narrow question: in the threads where a recommendation changes a purchase, who is doing the recommending?

New Knife Day is my site for knife collectors. It catalogs knives and steels and tracks which knives people on Reddit are buying and arguing about, so I have a stake in knife Reddit being worth reading. New Knife Day is at new.knife.day.

Nobody publishes their shill accounts, so I had to decide in advance what paid posting would leave behind. The market for it is not hidden. REDCmts sells one Reddit comment for $9.99 and 100 for $699.99, from what it calls "real, aged accounts", and shows a gallery of brand mentions it says it delivered. Soar says its accounts are "aged and manually warmed" for weeks before a single brand mention. Bazzly advertises automated replies to every post that looks like someone shopping.

Taking those sales pages as the description of the product, a paid campaign for one brand, delivered through a handful of prepared accounts, should show up as:

Every one of those is also what a devoted fan, a maker's employee posting on their own time, or a brand's own subreddit regulars wandering into a buying thread would produce. Public Reddit data can show that recommendations are concentrated, and cannot show why. Everything below is about the first half.

The scraper stores each new post and its comments shortly after posting. That turned out to be the wrong moment for this question: a buying thread collects its recommendations over the following day or two, and r/knives posts had 1.5 stored comments each. A refresh pass went back to 3,607 posts older than 48 hours and refetched their comment trees, which took the corpus from 21,673 comments to 51,129. Before the refresh the test below found nothing, because about 800 buying-thread mentions were too few to tell 7.6% from the 7.1% chance gives.

Three definitions do the work. A buying thread is a post whose title or body matches phrases like "should I buy", "recommend", "under $" or "best knife"; it is a regular expression, not a classifier. An author's brand-heaviness is the share of their comments that name a brand, weighted toward naming the same brand each time. The tail is the top 5% of the 987 authors with 10 or more comments on that score, 49 accounts. I score it this way because an account that keeps bringing up the same brand is the product the vendors above are selling.

The question is what share of buying-thread mentions the tail would write by chance, given how much everyone posts. To get that number I keep every brand mention where it is, in its thread and naming its brand, and reassign the author names at random across those mentions, like shuffling name tags. A prolific account still gets many mentions and a quiet one gets few, but which threads an account appears in no longer depends on who it is. I do this 1,000 times and record the tail's share each time. If the real share sits inside the range those 1,000 reassignments produce, the tail is no more concentrated in buying threads than its comment count explains. If it sits above the range, the tail accounts turn up in buying threads more often than their posting volume explains. Authors are salted hashes throughout; no usernames or comment text leave the database.

If the tail shows up in buying threads only because it posts a lot, random reassignment says it should write about 7.9% of the brand mentions there, and almost always between 6.3% and 10.1%. It wrote 11.3%. Only 2 of the 1,000 random reassignments reached that. In plain counts, about one buying-thread recommendation in nine comes from these 49 accounts, where chance says one in thirteen, which is about 50 extra recommendations out of 1,471.

Where the extra share lands matters more than its size. If knife Reddit as a whole were being gamed, every subreddit would sit above chance. Two do, r/chefknives and r/knifeclub. r/knives, the largest, is within half a point of chance. A paid campaign is bought by one brand, so it should also show up brand by brand, and it does: three brands get far more of their buying advice from the tail than chance gives, one large brand gets slightly less, and three get none. That is the shape a few targeted campaigns would leave, and also the shape a few loud fan bases would leave.

The brands are coded because a concentration statistic is not evidence that any brand paid for anything, and the brand with the strongest signal also has a large, loud fan base.

On corpus data, the tail also matches two more of the predictions above. The accounts are thin: a median of 12 comments in these six subreddits, and a median score of 1, so nobody is upvoting them into prominence. They are loyal to one brand: two thirds of a tail account's brand mentions go to the same brand. That is three of the six predictions, all the corpus can test. Age, knife focus and store links need each account's whole Reddit history.

I took the 23 tail accounts behind the three brands with a signal and fetched everything Reddit would return for each: up to about 2,000 comments and their submissions, account creation date and karma. For comparison, 23 other 10-plus-comment authors from the same corpus, drawn at random from the tail's comment-count range. Eight tail histories and seven comparison histories were hidden, suspended or deleted; the rest were readable.

A prepared paid account should be young, post mostly in the subreddits it is paid to post in, and push one brand. The tail accounts are 4.5 years old at the median, the same as the comparison accounts. Only 3% of their comments are in the six knife subs, against 10% for the comparison group, and they spread over 66 subreddits to the comparison group's 47. Across their whole history they name seven knife brands, not one. Store links are rare in both groups.

I compared about two dozen features in all. With 15 and 16 readable accounts, none of the differences is larger than splitting 31 people into two random groups would often produce, and the ones that lean anywhere lean toward the tail being less knife-focused.

That is not what a warmed, single-purpose account looks like. It is what a person who is on Reddit a lot, and who has strong feelings about one knife maker, looks like. It is also what a well-run paid account would look like, which is why the vendors sell aged accounts instead of fresh ones. The eight hidden tail histories could hold the whole story and I cannot read them.

Adding "reddit" to a knife search still gets you humans, mostly. For a couple of brands, a quarter to a third of the buying advice comes from accounts that mostly recommend that brand. Whether those are fans or paid, I do not know, and after reading their histories I lean toward fans and hold the lean loosely.

The practical check is the one the data endorses: when a knife recommendation comes from an account you do not recognize, click through and see whether it has ever named a different brand. In this corpus that one question separates the tail from everyone else better than karma does. It will not catch the paid comment from a real aged account, and I do not think anything a reader can do will.

Code, summary tables and charts are in the scraper repository linked below. Usernames, comment text, account histories and the brand-code key are not published and will not be.

I'm Peter Vijeh. Hiring, consulting, or you want to argue with this write-up: send it here and it lands in my inbox.