返回资讯中心
外部精选
软件工程
#开发者实时流

TrendsWorks报告:LLM分析师在5个议题上与数学聚类结果相左

来源标题:The LLM Overruled the Math and It Was Right

TrendsWorks发布2026年10月30日至11月5日趋势雷达报告,从12,243篇报道中聚类出225个话题,其中5个话题上LLM分析师与数学统计结果不一致。该系统刻意将数学聚类与带记忆的LLM分析师分开设计,允许二者给出不同判断。首个聚类案例中,数学方法将AI系统评估话题判定为“衰退”,z-score为−0.18,增长为近期均值的0.85倍。

Hacker News 最新lriesberg2026/11/10 · 7 天前30 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

This week’s report: Trends radar: October 30 – November 5, 2026 — 225 clusters from 12,243 stories, and 5 topics where the LLM analyst disagreed with the numbers.

Identifying trends that span weeks or months is a different problem than identifying topics that are trending today. Over time, topics evolve. They split, merge, go quiet, and then come back. The same idea gets new vocabulary, and the same word starts meaning something different. Counting keywords can’t follow that.

So TrendsWorks has two brains. The first is math: embeddings, clustering, and week-over-week counts that decide what’s emerging. The second is an LLM analyst with a memory. It reads those numbers, remembers what it said last month, and tells the story. I deliberately kept them separate, and I designed them to disagree.

On the very first run, the math had embedded about a thousand recent articles, grouped them into hundreds of clusters, and stamped a status on the 8 that qualified.

One of those clusters was a set of articles about evaluating AI systems. The math called it “fading” with a z-score of −0.18 and growth of 0.85× compared to its recent average. That’s a pretty clear verdict. The topic had a soft week and was on its way out.

But then the LLM analyst read the same numbers and overruled it:

The quantitative layer calls this cluster “fading” based on a z-score of –0.18, but the underlying articles span Gradient Flow and Eugene Yan and cover three distinct sub-arguments: evaluation is the real bottleneck (not model quality), AI operational bills are rising even as token prices fall, and agents need to be judged on task completion rather than capability demos. That is a coherent, durable storyline rather than a single viral piece decaying. Flagging as Emerging in the ledger rather than fading…

That wasn’t a bug, and it wasn’t the LLM going off the rails either. The system is designed so the two halves can contradict each other. Every disagreement is recorded with a reason, and the weekly report has a section just for exploring the disagreements. Neither layer gets the final word.

Nearly four months and 17 weekly runs later, “AI Systems Evaluation & Operational Cost” is still on the analyst’s ledger as Emerging. One soft week wasn’t decay after all.

Getting here took three wrong turns, and each one taught me something about the data. There’s a prompt at the end if you want to build something like this yourself. The scheme isn’t specific to articles. It should work on any stream of text with timestamps, such as forum posts, support tickets, papers, reviews, or commit messages.

Items come in every day and pile up in a SQLite database. Once a week, two very different layers look at that pile:

That’s the whole system. The analyst is responsible for carrying storylines across weeks, and the math is responsible for the numbers. The analyst can argue with the numbers but can’t change them.

Here’s what a disagreement looks like in the output. This one is from the September 11 run, where the analyst noticed that a cluster’s label didn’t match what was in it:

{ "subject": "Apple App Store policies and developer disputes (34 items, emerging, z 3.33)", "math_says": "A sizeable emerging cluster labeled around App Store policy and developer disputes.", "analyst_says": "Mostly a mislabeled Apple hardware-launch spike — the exemplars are iPhone 18 Pro, AirPods 5, Mac mini M6, iPhone Duo, not policy disputes.", "why": "The cluster label doesn't describe its contents; the mass is a September Apple-event product dump (a predictable calendar spike), with only one genuine App-Store-rejection item, so the 'developer disputes' framing overstates a real trend." }

It took a few wrong turns to get to that point.

The first clustering run used a cosine similarity threshold of 0.75, which is a pretty standard starting point. It produced one giant cluster of 715 items, about 70% of the week’s stories. Its first few members were “Israel’s $50M Experiment to Change U.S. Public Opinion,” “The World Cup Could Not Be Americanized,” “The Secret Origins of Amazon’s Alexa,” and “My Images Are Dithered.” Nothing in that list belongs together.

The problem was that I trusted a threshold before I’d measured the space it was operating in. So I ran a quick test on two sentences that have nothing to do with each other:

from shared.embeddings import embed_texts

a, b = embed_texts([ "DuckDB adds columnar json support", "A recipe for sourdough bread", ]) print(a @ b) # 0.595

Databases and sourdough came out at 0.595. With this embedding model (gemini-embedding-001), unrelated text sits around 0.6. A 0.75 threshold is barely above noise, which is why nearly anything with “AI” in it ended up in the same bucket.

So I swept the threshold over a week of stories (about 1,000 items):

The numbers alone don’t tell you which row is right. For that, I wrote a small script that prints the contents of each cluster at a given threshold, and then I just read them. At 0.85, the best clusters snapped into single topics. One was the launch of a new Claude model: “Claude Opus 5”, “Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard”, “Elevated errors on Claude Opus 5”. At 0.88, related articles started splitting apart, and 94% of clusters held a single story.

--- cluster of 23 --- [hn] Claude Opus 5 [hn] Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard [hn] "We removed over 80% of Claude Code's system prompt for Opus 5 and Fable 5" [hn] ARC-AGI Leaderboard [hn] Claude Code has a hardcoded instruction telling Opus 5 not to use subagents [hn] Elevated Errors for Opus 5 ... and 17 more

There’s no metric here. The evaluation is me reading clusters and deciding whether they belong together. At this scale, that’s fine. Someone who knows the domain can look at 30 clusters in ten minutes and tell whether they’re coherent, and that’s a better check than any metric I could have created.

This came up again later. On their own, titles are thin, so I changed what gets embedded. Now it’s the title plus the top comments, which say a lot more about what a story is actually about. When I re-ran the calibration, the best threshold was still 0.85, but the clusters at that threshold were a lot better. The big catch-all blob split out into several coherent themes. It was the same number, but different input, and I knew it still held because I checked.

Window size matters too. Run the same 0.85 over three months (about 11,000 items) and the largest cluster grows from 114 to several hundred, because a greedy algorithm that keeps absorbing near neighbors drifts as more stories arrive. That’s one reason the analyst reads cluster contents and doesn’t trust their labels.

With clustering fixed, the first run’s strongest signal (z = 1.36, growth 3.14×) turned out to be… one prolific author writing about PostgreSQL. Great posts, but one person publishing on their own schedule isn’t a trend. The analyst spent a big chunk of that first report explaining why single-source clusters don’t count.

The next strongest signals were two clusters built entirely from one vendor’s product blog. The analyst threw those out too:

These are product release notes, not trend signal. A vendor announcing GreyNoise integration or SOC 2 certification does not constitute a community storyline.

It was right and the problem wasn’t the model. It was the data. At that point, the corpus was mostly RSS feeds from about 200 blogs. Blogs are great context, but trends need many independent people reacting to the same things, with real timestamps. A feed of 200 personal blogs is 200 sample sizes of one.

So I changed the main source to the Hacker News firehose and pulled every story that gets at least 10 points, which works out to about 150 a day. I also dropped the keyword filtering.

The original version of the HN agent searched for keywords like “pandas,” “sql,” and “machine learning,” which seems sensible, but it means the system could only ever find things I already knew to look for. If I’m building a system to judge quality, filtering for precision makes sense. For a radar, I want coverage. My filters were encoding my blind spots.

Another advantage of using Hacker News data is Algolia indexes all of it, including timestamps, going back years. I backfilled 12 months of history in minutes. The corpus went from about 4,300 mostly-blog items to about 61,000 today.

This one is less dramatic but it’s the gotcha most people will hit.

Hacker News has a memory. You can ask Algolia for everything posted last March and get it back with real timestamps. You can’t do that with RSS. A feed shows you what’s in it right now, and once a post scrolls off the end, it’s gone for good. That’s why the TrendsWorks system keeps its own corpus. If you can’t go back and get a source’s history, you have to collect it as it happens.

And once you’re collecting, you need a clock. Every item gets a discovered_at timestamp the first time the system sees it, and that value never changes. If the same story shows up the next day with more points, only the engagement numbers get updated.

That sounds like a small detail, but if re-seeing an item bumped its timestamp, every popular story would smear across several weeks and the weekly counts would be wrong. discovered_at is the trend clock, and the whole math layer depends on it being right.

Then the backfill nearly broke it. Backfilled items need discovered_at set to the story’s original published time. If you naively stamp them with “now,” a year of history collapses into a single day. In my case it also tripped a sanity rule that was meant to catch RSS feeds that resurface ancient posts. Neither failure is obvious until you look at a histogram with one enormous bar in it.

There’s one more time rule worth mentioning: the math only analyzes complete weeks. The report goes out Friday morning and covers Friday through Thursday night. The week that’s in-progress is excluded, so the system never mistakes “the week isn’t over yet” for “this trend is dying.”

There are two choices here that people will want to argue with.

No vector database. Embeddings are stored as BLOBs in SQLite. The pipeline never does live nearest-neighbor lookups. It clusters one window, once a week, in one batch, and brute-force cosine similarity over the whole thing is a few milliseconds of numpy at this scale. A vector database would be more infrastructure to run for a search problem this system doesn’t have.

No HDBSCAN. The clustering is a greedy centroid algorithm in about 40 lines. It has one knob, the threshold from Failure #1, and every assignment has a one-sentence explanation: “this item was at least 0.85 similar to that cluster’s centroid.” Here’s the whole thing:

def cluster_items(vectors: np.ndarray, threshold: float = 0.85) -> list[Cluster]: """Assign each vector (rows L2-normalized, chronological order) to the most similar existing cluster if cosine ≥ threshold, else start a new one. Returns clusters ordered by first appearance.""" clusters: list[Cluster] = [] if len(vectors) == 0: return clusters centroids = np.empty((0, vectors.shape[1]), dtype=np.float32) sums = np.empty((0, vectors.shape[1]), dtype=np.float32) # unnormalized member sums

for i, vec in enumerate(vectors): if len(clusters) > 0: sims = centroids @ vec best = int(np.argmax(sims)) if sims[best] >= threshold: clusters[best].member_indices.append(i) sums[best] += vec c = sums[best] / np.linalg.norm(sums[best]) centroids[best] = c clusters[best].centroid = c continue

start a new cluster

clusters.append(Cluster(centroid=vec.copy(), member_indices=[i])) centroids = np.vstack([centroids, vec[None, :]]) sums = np.vstack([sums, vec[None, :]])

return clusters

A fancier algorithm could probably produce slightly better clusters, but the whole point of this system is that the analyst (and I) can argue with the math, and you can’t argue with something

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →