返回资讯中心
外部精选
软件工程
#开发者实时流

Measuring Intelligence Beyond Human Scale

This pattern was first formalized by Charles Spearman in 1904, an observation known as the positive manifold. To explain this regularity, he proposed the idea of a common latent factor, which he coined “general intellig…

Hacker News 最新nicholascarolan2026/10/19 · 2026/10/199 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

This pattern was first formalized by Charles Spearman in 1904, an observation known as the positive manifold. To explain this regularity, he proposed the idea of a common latent factor, which he coined “general intelligence (g)”.[3]

Spearman’s work1 has shaped modern psychometrics by presenting intelligence not simply as a collection of separate observable capabilities, but as an underlying statistical variable inferred from the patterns of observable behavior. Interestingly, this is also the origin of factor analysis in statistics.

Classically, intelligence has been measured through absolute performance on specific tasks, which are curated by human experts. In particular, a task solved by all participants or a task too difficult for all participants is uninformative about their relative abilities. In the case of artificial systems, however, systems can have capabilities approaching or exceeding those of the experts designing the examination.

One potential solution is to measure the absolute measurement with a relative comparison through the utilization of a protocol that makes judgment dependent on the performance of a subject relative to another. Louis Leon Thurstone formalized this idea in his 1927 Law of Comparative Judgement[4], that repeated pairwise competitions can be used to recover an underlying latent scale. A familiar practical example is the Elo rating system, in which the relative abilities of Chess or Go players are estimated through pairwise games. This approach is especially useful when the frontier of capability is unknown; a fixed standard need not be defined in advance, and metrics can scale as stronger participants enter the population.

A natural approach, based on Turing’s intelligence definition, is a pairwise protocol where two agents alternate between proposing problems and solving those posed by their opponent. Proposers are rewarded for producing valid problems the solver cannot solve, while solvers are rewarded for correct solutions. Results across rounds can then be aggregated using Bradley-Terry style ratings.

Recent work applies this evaluation protocol in verifiable domains. The Token Games[5] evaluates models through executable logic puzzles with deterministic verification and Elo-based rankings. Similarly, MathDuels[6] evaluates models as both problem authors and solvers using independent verification.

The main limitation is adjudication. Restricting challenges to mathematics, coding, or other verifiable tasks simplifies evaluation but excludes many open-ended forms of reasoning. A protocol intended to remain useful as models become more capable cannot assume that all meaningful questions are mechanically verifiable.

Using another model or a panel of models as a judge merely shifts the problem. Judges must be at least as capable as the systems they evaluate and robust to manipulation. Strong problem-solving ability also does not imply reliable judgment. Model-family biases, stylistic cues, or hidden signals may also influence decisions, especially in an adversarial setting.

Requiring the proposer to commit to an answer is likewise insufficient. A proposer may lie or exploit private information or computational complexity by creating challenges whose difficulty reflects information or resource asymmetry rather than reasoning ability. For example, the proposer may ask questions that have private information, such as:

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →