返回资讯中心
外部精选
软件工程
#开发者实时流

TopoVoid-0.2B 发布:在概念零空间中直接写入模型权重

来源标题:Show HN: TopoVoid-0.2B – Closed-form weight writing in concept null space

TopoVoid-0.2B 在 Hugging Face 发布,采用闭式解在概念零空间中写入权重,用于知识编辑与概念瓶颈研究。模型支持文本生成与分类,覆盖 AG News、DBpedia-14 等 4 个数据集,许可证为 Apache-2.0。

Hacker News 最新Tirpitzia2026/11/3 · 14 天前30 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

Like 2

Follow

Topovoid 2

Text Generation

PyTorch

Safetensors

4 datasets

English

Chinese

topovoid

concept-bottleneck

knowledge-editing

null-space-projection

License: apache-2.0

Model card Files Files and versions xet

Community

Copy to bucket

AG News</code> & <code>DBpedia-14</code>)","children":[],"isValid":true,"title":"2.4 Text Classification (<code>AG News</code> & <code>DBpedia-14</code>)"},{"id":"25-external-midcbm-retrieval-index--copy-head","label":"2.5 External MidCBM Retrieval Index + Copy Head","children":[],"isValid":true,"title":"2.5 External MidCBM Retrieval Index + Copy Head"}],"isValid":true,"title":"2. Quick Start & Open-Dataset Benchmark"},{"id":"3-further-research--next-gen-multimodal-results","label":"3. Further Research & Next-Gen Multimodal Results","children":[],"isValid":true,"title":"3. Further Research & Next-Gen Multimodal Results"},{"id":"4-limitations","label":"4. Limitations","children":[],"isValid":true,"title":"4. Limitations"},{"id":"5-license--attributions","label":"5. License & Attributions","children":[{"id":"license","label":"License","children":[],"isValid":true,"title":"License"},{"id":"attributions","label":"Attributions","children":[],"isValid":true,"title":"Attributions"},{"id":"citation","label":"Citation","children":[],"isValid":true,"title":"Citation"}],"isValid":true,"title":"5. License & Attributions"}],"isValid":true,"title":"TopoVoid-0.2B-Orthogonal-Preview"}]}"> TopoVoid-0.2B-Orthogonal-Preview 1. Overview Architecture

Files in This Repository

  1. Quick Start & Open-Dataset Benchmark Summary of Open-Dataset Results

2.1 CounterFact (1-Hop Weight Write)

2.2 Stacking Modules

2.3 MQuAKE-CF-3k (2-Hop)

AG News</code> & <code>DBpedia-14</code>)">2.4 Text Classification (AG News & DBpedia-14)

2.5 External MidCBM Retrieval Index + Copy Head

  1. Further Research & Next-Gen Multimodal Results

  2. Limitations

  3. License & Attributions License

Attributions

Citation

TopoVoid-0.2B-Orthogonal-Preview

Organization: Topovoid (topovoid.com)

Maintainers: tirpitzia, 0kaguya

License: Apache-2.0

  1. Overview

This repository contains the 0.2B research preview (v0.1) of the TopoVoid architecture. It is a 212M-parameter bilingual causal language model pretrained from scratch with explicit middle-layer concept bottlenecks, paired with a closed-form solver that writes new facts into Attention and MLP weights inside the orthogonal null space of those concept layers.

How the closed-form write works (closed_form_write.py):

Architecture

Item Value

Parameters 212,011,264 (tied input and output embeddings)

Layers, d_model, d_ff 16, 1024, 2432 (SwiGLU)

Attention 16 query heads, 4 KV heads (GQA), per-head QK-RMSNorm

Vocabulary 49,152 active tokens, pruned from 151,936 (~82M unused embedding parameters removed)

Positional encoding RoPE (theta = 5e5, 4,096-position table)

L7 MidCBM 24 dimensions (35:1 concept SNR design)

L7 SubspaceTopKCBM 40-d subspace, 256 dictionary atoms, Top-8

L12 Highway CBM 24 dimensions

Files in This Repository

File Description

model.safetensors Pretrained fp32 weights (809 MB); head.weight is tied to emb.weight on load

config.json, tokenizer.json, tokenizer_config.json Model config and Qwen-based fast tokenizer

vocab_map_49152_chatml.npz Active-vocabulary index map (orig_to_compact, compact_to_orig) and 64 × 256 calibration token rows (eval_c4, eval_zh)

modeling_topovoid.py PyTorch model definition and load_topovoid helper

closed_form_write.py Closed-form null-space weight writer ($\Delta W = P_\perp R V$)

demo_write.py Minimal verification script (subspace split, closed-form write, cosine retention, and before/after generation)

  1. Quick Start & Open-Dataset Benchmark

Tested with torch 2.8.0, transformers 4.57.1, safetensors 0.8.0, and numpy on one GPU.

from modeling_topovoid import load_topovoid from closed_form_write import concept_null_projector

model, tok, o2c, c2o = load_topovoid('.') P_perp, Q_cbm = concept_null_projector(model)

Run the minimal verification and closed-form write demo (demo_write.py), or pass --counterfact to evaluate batch compilation on counterfact.json:

python demo_write.py python demo_write.py --counterfact counterfact.json --n 12

Summary of Open-Dataset Results

The three evaluation tracks below test distinct mechanisms—direct weight writing, frozen-feature linear readout, and external concept-index retrieval—and should not be conflated:

Example raw output on a CounterFact edit (target Toronto, greedy decoding, max 16 new tokens):

Prompt : Sunrise Party is headquartered in Before write : the heart of the city. The club is a 100% free After write : Toronto, Canada. The company is based in Toronto, Canada, and is a

2.1 CounterFact (1-Hop Weight Write)

First edit layer: Layer 8 (1-based). CounterFact metrics below use first-target-token proxies of ES / PS / NS (official ROME scores full target sequences):

2.2 Stacking Modules

Each module contains 40 CounterFact facts compiled independently from the same base checkpoint and added linearly (W + Σ ΔW_i):

Note on multi-module decay and the 1B compiler upgrade: In this 0.2B preview repository, closed_form_write.py only projects each module individually against the protected concept basis $Q_{\mathrm{cbm}}$ ($P_\perp = I - Q_{\mathrm{cbm}} Q_{\mathrm{cbm}}^T$) and does not include a cross-module Gram-Schmidt joint orthogonalization operator across different modules' right-hand key subspaces. As a result, 2 stacked modules (80 facts) retain 96.25% Top-1 (0.997886 Highway cosine), whereas summing 4 or 8 independently solved modules causes key collisions inside the null space (55.62% and 34.69%). Cross-module Gram-Schmidt joint compilation is implemented in our internal 1B commercial compiler (see Section 3).

2.3 MQuAKE-CF-3k (2-Hop)

Evaluated on 150 sampled cases (149 allow the unrelated-subject control). These numbers do not establish multi-hop reasoning on the base checkpoint:

2.4 Text Classification (AG News & DBpedia-14)

A separate ridge classifier head is fitted on frozen representations without changing the language model weights or generation path. Full L7 (1024-d) and CBM (320-d, MidCBM + SubspaceTopKCBM) are reported separately; Full L7 results cannot be attributed to the concept bottleneck alone.

Fixed-Hyperparameter 40-Label Test (AG News, 10 Labels/Class, 0 Validation Labels)

With ridge $\lambda = 0.1$ and LinearSVC $C = 1.0$ fixed in advance across 5 sampling seeds (20261203–20261207) and no validation labels used for selection:

Five-Seed Low-Label Curve (With 8,000-Example Validation Selection)

Each method uses the exact same training subsets (seeds 20261203–20261207) and selects hyperparameters on the 8,000-item validation split:

Full-Dataset Readout & Perturbation Checks

AG News uses 112,000 fit / 8,000 val / 7,600 test items. DBpedia-14 uses 120,000 fit / 8,000 val / 70,000 test items (10,000-item test subset for zero-shot rows; on that same 10,000-item subset, CBM / Full L7 / TF-IDF score 90.96% / 96.53% / 98.23%):

Under surface perturbations on AG News (evaluated without retraining the head): Original 89.00% (TF-IDF 91.82%), All lowercase 86.45% (91.82%), Strip Reuters/AP datelines 88.83% (91.45%), First 32 tokens 86.09% (90.75%). The 4-class L7 head is 40 KiB (32 KiB float64 weights + 8 KiB float32 mean/std). Full-split tokenization takes 15.89 s, feature extraction 373.46 s, and ridge fitting 1.75 s per $\lambda$ candidate.

2.5 External MidCBM Retrieval Index + Copy Head

On 399 unseen paraphrases of stored ARC science questions, indexing frozen MidCBM and SubspaceTopKCBM states in an external retrieval index with a zero-parameter copy head raises accuracy from 28.57% (0.2B alone) to 72.93% (+44.36 pp, 95% CI [+39.36, +49.38] pp; 72.68% with shuffled options; 96.3% precision when triggered during 24-token greedy generation; 1.000009 max PPL ratio on C4/ZH).

  1. Further Research & Next-Gen Multimodal Results

Based on this early 0.2B orthogonal framework, our team has completed further internal work on cross-modal representation alignment and industrial compilation toolchains (including cross-module Gram-Schmidt joint orthogonalization), scaling this formulation to 1B / 3B / 7B multimodal architectures.

On our 1B model (currently in testing and rollout across production workflows), mounting closed-form orthogonal modules without fine-tuning base weights yields:

Technical details and industrial pipelines (CAD parsing and Tech Pack breakdown) are available at https://topovoid.com.

  1. Limitations

Readout heads are separate modules. A classification head reads L7 features and outputs class scores directly. It does not change what the language model generates and does not show general reasoning gains.

Cosine 1.0 on L7 MidCBM comes from causal order as well as projection. With the first edit at Layer 8, L7 MidCBM and SubspaceTopKCBM activations are computed upstream of the edited weights and cannot change. The L12 Highway CBM is downstream of layers 8–11 and is not strictly invariant (0.9979 → 0.9890 mean cosine across 2 to 8 stacked modules).

Module stacking interferes beyond 2 modules in v0.1. In this 0.2B preview, closed_form_write.py only performs single-module orthogonal projection against $Q_{\mathrm{cbm}}$ ($P_\perp = I - Q_{\mathrm{cbm}} Q_{\mathrm{cbm}}^T$) without a cross-module Gram-Schmidt joint orthogonalization operator. While 2 stacked modules (80 facts) retain 96.25% Top-1, adding 4 or 8 independently compiled modules degrades Top-1 to 55.62% and 34.69% and raises perplexity. Cross-module joint orthogonal compilation is part of our internal 1B commercial compiler.

Report paraphrase and neighborhood metrics beside rewrite Top-1. Rewrite Top-1 measures reproduction of the written prompt. Paraphrase generalization is lower (35.0%–41.7% Top-1), and edits bleed into neighboring subjects of the same relation (27.5%–56.7%).

Multi-hop hits require the control. On MQuAKE-CF-3k, 117/150 multi-hop questions hit the edited answer, but 118/149 unrelated-subject control questions also trigger the same answer via surface question templates.

Writes raise perplexity on unrelated text, scaling with the number of facts written into a single module.

The external memory index only recalls stored items, and can copy a wrong stored answer when retrieval is confident but incorrect.

Pruned vocabulary. Unsupported tokenizer IDs map to compact ID 0 in ordinary forward passes; closed_form_write.py splits targets into active subtokens where possible.

Base model without safety tuning. Pretraining stages used web text without spam or adult-content filtering, and no chat or safety alignment was applied. Do not deploy directly in user-facing production.

  1. License & Attributions

License

Code and TopoVoid-0.2B-Orthogonal-Preview weights in this repository are released under the Apache License 2.0 (see LICENSE).

Attributions

Tokenizer and distillation teachers: Qwen tokenizer and Qwen teacher models (Qwen, Apache-2.0)

Pretraining and calibration text: FineWeb, FineWeb2 (ODC-By 1.0), and C4 (ODC-By 1.0) (eval_c4, eval_zh in vocab_map_49152_chatml.npz)

Evaluation benchmarks: CounterFact (MIT), MQuAKE (MIT), AG News, DBpedia-14 (CC-BY-SA-3.0), and AI2 ARC (CC-BY-SA-4.0)

Citation

@misc{topovoid2026orthogonal, title = {TopoVoid-0.2B-Orthogonal-Preview: Native Orthogonal Subspace Architecture and Closed-Form Null-Space Weight Compilation}, author = {tirpitzia and 0kaguya and {TopoVoid Research}}, year = {2026}, howpublished = {\url{https://huggingface.co/Topovoid/TopoVoid-0.2B-Orthogonal-Preview}}, note = {Research Preview v0.1, \url{https://topovoid.com}} }

Downloads last month 5

","eos_token":"<|endoftext

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →