返回资讯中心
外部精选
软件工程
#开发者实时流

Itzma 发布:基于对象存储的嵌入式向量数据库,用 Rust 编写并提供 Python 绑定

来源标题:Itzma, an embedded vector database backed by object storage

Itzma 是一个以对象存储为后端的嵌入式向量检索库,用 Rust 编写并提供 Python 绑定,向量以普通对象形式存入 bucket,调用方与数据之间无需服务器。它支持 L2、cosine、dot product 三种度量,默认使用 HNSW 索引,也提供 SQ8 聚类分区索引,并支持稀疏向量与类型化属性过滤。写入通过 put-if-absent 和条件更新保证正确性,支持多写入者批量 upsert 和删除,并兼容 S3、本地磁盘与内存存储。

Hacker News 最新Kh40sz3r02026/11/16 · 2 天前30 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

Itzma is vector retrieval on top of object storage, written in Rust, with Python bindings. Vectors are stored as plain objects in a bucket. There is no server between the caller and the data: any process that can reach the bucket can write, and any process can query. Correctness rests on two things the object store already does, put-if-absent and conditional update.

It is a library that runs in your process. Open a namespace, and S3 (or a local directory, or memory) holds the data.

Namespaces with a fixed dimension and metric (L2, cosine, dot product).

Dense vector search, exact or through an index: HNSW (the default) or SQ8 with clustered partitions.

Sparse vectors of (term id, weight) pairs beside the dense one. Sparse queries are exact, and the inverted index is optional.

Typed attributes declared at create. Filter on them alone, or use them to narrow a vector query.

Upserts and deletes go in batches, and many writers can send them at once. Idempotency keys are optional.

S3 (and anything else object_store supports), local disk and in-memory stores.

Compaction, garbage collection and manifest recovery.

A RAM and NVMe read cache and a cache of decoded structures.

Rust and Python APIs; NumPy and Arrow input in Python.

A read-only health report and a byte budget on every read.

Quick start

Rust: add itzma-core from crates.io. Python: pip install itzma (wheels for macOS and Linux, Python 3.10 and later; see py-itzma). This example uses the in-memory store.

[dependencies] itzma-core = "0.1" object_store = "0.14" tokio = { version = "1", features = ["macros", "rt-multi-thread"] } serde_json = "1"

use std::sync::Arc;

use itzma_core::{AttrSchema, AttrType, Document, Metric, Namespace, Schema}; use object_store::memory::InMemory;

#[tokio::main] async fn main() -> itzma_core::Result<()> { let store = Arc::new(InMemory::new());

// Dimension, metric and the type of every attribute are fixed for the life // of the namespace. let schema = Schema::new(3, Metric::Cosine) .with_attrs(AttrSchema::new().field("title", AttrType::Text)); let ns = Namespace::create(store.clone(), "articles", schema).await?;

// One call is one atomic write. It returns the sequence number. let seq = ns .upsert(vec![ Document { id: "intro".into(), vector: vec![0.9, 0.1, 0.0], attrs: Some(serde_json::json!({ "title": "Introduction" })), sparse: None, }, Document { id: "faq".into(), vector: vec![0.0, 1.0, 0.2], attrs: None, sparse: None, }, ]) .await?; assert_eq!(seq, 1);

// Closest first. Lower distance means closer, for every metric. let hits = ns.query(&[1.0, 0.0, 0.0], 2).await?; assert_eq!(hits[0].id, "intro");

ns.delete(vec!["faq".into()]).await?;

// Another handle, possibly in another process, sees the same data. let other = Namespace::open(store, "articles").await?; assert_eq!(other.query(&[0.0, 1.0, 0.0], 5).await?.len(), 1); Ok(()) }

The same in Python (build it with make py-dev, see py-itzma/README.md):

import itzma from itzma import attr

store = itzma.Store("memory://") # or "file:///var/lib/itzma", "s3://bucket/prefix" schema = itzma.Schema(dim=3, metric=itzma.Metric.COSINE, attrs={"year": itzma.AttrType.NUMBER}) ns = itzma.Namespace.create(store, "articles", schema)

ns.upsert([ {"id": "intro", "vector": [0.9, 0.1, 0.0], "attrs": {"year": 2024}}, {"id": "faq", "vector": [0.0, 1.0, 0.2], "attrs": {"year": 2021}}, ]) hits = ns.query([1.0, 0.0, 0.0], top_k=2, filter=attr("year") >= 2022) print(hits.ids, hits.distances)

Writes also take NumPy arrays (ids=, vectors=, attrs=) and Arrow, Polars or DuckDB tables.

Namespace. A collection of documents that share a schema. On storage it is a key prefix, <id>/. Ids match [a-z0-9_-]{1,64}. A namespace is created explicitly with Namespace::create, since its dimension and metric must be declared first.

Document. An id (any string of 1 to 512 bytes), a vector and optional attrs, a JSON object whose fields the schema declares. Upserting an existing id replaces the whole document.

Schema. { dim, metric, attrs }. Every vector written or queried must have exactly dim finite components; under cosine an all-zero vector is refused too. The schema lives in the manifest and again in schema.json, which is written once and never changed, so losing the manifest does not lose it.

WAL entry. One upsert or delete call becomes one entry in the write-ahead log, committed as a whole. Entries are numbered 1, 2, 3, and so on.

Manifest. The one object that is ever overwritten. It holds the schema, the live segments, how far the WAL has been folded into them, and which segments have an index.

Segment. An immutable object holding the live documents and deletions of a range of WAL entries, named by that range. A query reads the segments plus the WAL entries not folded yet (the tail).

Handle. A Namespace value holds the store, the id, the schema and a few hints. It holds no lock, so any number of handles, in one process or many, can work on one namespace.

upsert(docs) and delete(ids) validate everything first, so one bad document refuses the whole call with nothing written. Then the batch is written as one object with put-if-absent at the next sequence number. If another writer took that number, the writer looks past it and tries again, up to 32 times (WriteContention). There is no coordinator, lock or leader.

A batch is one object and is refused above 64 MiB uncompressed (BatchTooLarge). Split bigger loads into several batches; each is atomic, the whole is not.

A write that fails after the request was sent may or may not have landed. A plain retry can append it twice, which is harmless for the end state of upserts and deletes but can overwrite a newer write to the same id. For loads that must land once, use the keyed forms, upsert_keyed and delete_keyed, with a WriteKey: WriteKey::new(key) on the first attempt, WriteKey::retry(key, from) on a retry, which looks for the key from sequence from on and returns the original's sequence number if it finds it.

Columnar input: upsert_columns takes a DocColumns (ids, a row-major f32 matrix, typed attribute columns) as one batch, and upsert_stream takes a stream of them, cut into objects of about StreamOptions::target_bytes and written in order. A failed stream reports how much landed. The Python bridge uses these for Arrow input.

query(vector, top_k) returns up to top_k hits, closest first, each with its id, distance and attributes. query_with takes QueryOptions:

Each segment with an index proposes candidates through it; segments without one and the WAL tail are scanned. A candidate that a newer segment or the tail overwrote or deleted is dropped, the survivors are reranked on exact vectors, and the closest top_k across all sources win. The answer never carries an approximate distance.

Distances: L2 is squared Euclidean, cosine is 1 - cos, dot product is negated, so lower is closer for all three. Distances are comparable only within one query.

Attribute fields are declared at create, each with a type: Bool, Number, Text, TextSet or NumberSet. Writes are checked against the schema: an undeclared field is UnknownAttr, a wrong type AttrTypeMismatch. null is allowed for any field.

Predicate is a typed tree: And, Or, Not, null and boolean tests, number comparisons, In lists and ranges, text comparisons, and set tests (Contains, ContainsAny, ContainsAll, IsEmpty). A comparison against a null or missing field is false, and Not flips it.

Namespace::filter(&predicate, &FilterOptions) returns matching rows in id order, a page at a time (limit 1 to 10,000, default 100; pass the returned next as after for the next page). It reads no vectors. Paging is not a snapshot.

A filter on query_with is applied after shadowing, and a source short of matches is asked again for more, so the answer is the true filtered top top_k, short only when fewer documents match.

Namespace::index(&IndexOptions) builds an index for each live segment with at least min_rows rows (default 4,096) that has none yet, and publishes it in the manifest. Neither compact nor maintain builds indexes, so schedule index after them (the Python maintainer can run it for you). IndexOptions::kind picks the kind; a namespace can hold both, and a rebuild can switch one segment's kind.

HNSW (IndexKind::Hnsw, the default) is a graph over one-byte-per-dimension codes. A query fetches a segment's whole index object in one request, then walks the graph out of those bytes, so it is cheap once the object is warm in a cache and costs one larger read cold. It takes longer to build than SQ8. At 32 segments it gave recall@10 of 0.987 on SIFT1M and 0.984 on GloVe-25.

SQ8 (IndexKind::Sq8) stores each vector as one byte per dimension and reads only the partitions a query probes, which suits uncached reads. Segments of 100,000 rows or more are clustered into about sqrt(rows) partitions (IndexOptions::nlist overrides), and a query reads about 2 * sqrt(nlist) of them by default. On SIFT1M and GloVe-25 the defaults gave recall@10 of 0.99 or more. rebuild: true builds again over existing indexes. An index needs segment_format 2 or above.

A document can carry a sparse vector beside its dense one: Document::sparse, a list of (term id, weight) pairs, optional per document, nothing declared per namespace. Term ids must not repeat and weights must be finite; at most 65,535 terms per document.

Namespace::query_sparse(&terms, top_k, &SparseQueryOptions) is exact and standalone: the score is the dot product, Hit.distance its negation, ties broken by id. It takes a filter like query_with. Shadowing across segments and the tail works as for dense queries.

Namespace::index_sparse(&IndexSparseOptions) builds an inverted index per segment (sparse-idx-<n>.bin), so a query reads only its own terms' postings instead of every sparse vector. The answer is the same with or without it.

A batch with a sparse vector is written as WAL version 4 (5 with a key), which needs the namespace at WAL level 5. New namespaces start there; an older one needs upgrade_wal_format(5).

compact folds the WAL tail into a new segment and publishes it with one conditional manifest update. When there are more than max_segments (default 8) live segments, or one is at max_segment_bytes (default 128 MiB), it merges instead, into size-capped segments, streaming the inputs. A segment's bytes depend only on its WAL range, so a rerun after a crash writes the same object, and racing compactors cannot publish different answers. Compaction never deletes anything.

gc deletes segments that a merge or a later fold replaced, once they and their replacement are at least min_age old (default 24 hours), so readers still holding an old manifest have time to finish. With fence_wal: true it also replaces folded WAL objects with 8-byte fences, which keep put-if-absent working at their keys. WAL objects are never deleted. dry_run reports what it would do.

maintain is the one call to schedule (from cron, a loop, a worker): it compacts when the tail is long enough (64 objects, 64 MiB, or an optional age) or there are too many segments, then runs gc. The library starts no background task itself; the Python bridge has an optional maintainer.

recover writes a new manifest for a namespace that lost its own, from schema.json, the segment names and the WAL. It refuses if a manifest exists, and stops with Unrecoverable and the keys to restore if a range has no readable copy. It does not restore indexes; run index and index_sparse again.

Format levels. The manifest records the newest WAL and segment formats the namespace may hold. A build that reads less refuses the namespace before touching it. upgrade_wal_format(level) and upgrade_segment_format(level) raise a level, one way. New namespaces start at WAL level 5 and segment level 2; upgrade_segment_format(3) turns on the typed attribute columns of segment format 3.

Read cache. CacheStore wraps any store with a RAM tier and an optional NVMe tier, keyed by obj

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →