Show HN: Find duplicates in 5 lines of Python (offline, lightweight, fast, Rust)

Compare anything. Keep every record clean.

Find duplicates in Python data, from text datasets to structured records. Combine fuzzy matching, local embeddings and field-level rules. Fast, lightweight, no server required.

pip install univo

from univo import Finder

names = ["Northwind Traders", "Contoso", "Northwnd Traders", "NORTHWIND TRADERS", "Fabrikam"] for match in Finder().find(names, min_score=0.8).matches: print(names[match.left_id], "~", names[match.right_id], round(match.score, 2))

Northwind Traders ~ NORTHWIND TRADERS 1.0 Northwind Traders ~ Northwnd Traders 0.94 Northwnd Traders ~ NORTHWIND TRADERS 0.94

No training and no API key: records never leave your machine.

By default, find returns up to 1,000 matches scoring 0.8 or more (limit, min_score), and text is compared by spelling. Text(mode="semantic") compares by meaning, with a local model: the 15 MB multilingual default, or any Model2Vec model from Hugging Face or a folder. More: choosing a mode, choosing a model, every option.

Built for data cleaning, entity resolution, ML dataset deduplication and real-time duplicate checks.

A schema says which fields to compare, how, and which differences rule out a match. Nested fields are reached by path.

from univo import Exact, Finder, Schema, Text

schema = Schema( id_field="id", fields={"name": Text(), "address.city": Text(), "vat": Exact(conflict="reject")}, ) finder = Finder(schema=schema) finder.add( [ {"id": 1, "name": "Northwind Traders SAS", "address": {"city": "LYON"}, "vat": "FR401"}, {"id": 2, "name": "Northwind Traders", "address": {"city": "Lyon"}, "vat": "FR999"}, ] ) query = {"id": 3, "name": "Northwind Traders", "address": {"city": "Lyon"}, "vat": "FR401"} for match in finder.search(query).matches: print(match.right_id, round(match.score, 2), match.reasons)

1 0.94 ['name: trigrams', 'address.city: edit_distance', 'vat: exact']

Record 2 has another VAT number, so it can never match.

from univo import Finder, Schema, Text

finder = Finder( schema=Schema(fields={"description": Text(mode="semantic")}), model="minishlab/potion-base-8M", # any Model2Vec model on Hugging Face, or a folder ) texts = ["online clothing store", "shop selling clothes on the internet", "steel factory"] report = finder.find([{"description": text} for text in texts], min_score=0.6) for match in report.matches: print(texts[match.left_id], "~", texts[match.right_id], round(match.score, 2))

online clothing store ~ shop selling clothes on the internet 0.74

Without model, Univo uses its 15 MB multilingual default. The model downloads once, is verified, then works offline (models).

Records, not just strings. Dicts, lists and nested fields, compared field by field, with an explanation for every match.

Rules before scores. Different VAT numbers, tenants or environments rule out a match, however alike the records look.

Spelling, meaning and exact values, chosen per field, with a 15 MB multilingual model that runs offline, or any Model2Vec model you choose.

Fast, exact by default, and light. 100,000 names deduplicated in 4.5 s. Search is exact by default, with explicit limits and optional approximate retrieval. One 2 MiB wheel, no Python dependency.

New records as they arrive

Keep an index and check each new record before you store it.

finder = Finder( schema=Schema(id_field="id", fields={"name": Text(), "vat": Exact(conflict="reject")}) ) finder.add([{"id": 1, "name": "Northwind Traders", "vat": "FR401"}])

finder.check_or_add({"id": 2, "name": "Northwnd Traders", "vat": "FR401"}).status # "review" finder.check_or_add({"id": 3, "name": "Contoso", "vat": "FR777"}).status # "inserted" finder.check_or_add({"id": 4, "name": "Northwind Traders", "vat": "FR401"}).status # "matched"

review means a similar record exists and nothing was inserted: your application decides.

Checking one new record against 100,000 synthetic company names, in one partition, at the default threshold of 0.8, with the model already loaded (Apple M2, PyPI wheel 1.0.1, script):

Times depend on your data: they grow with the records of a partition and with the number of close candidates. Indexes can be updated, saved and loaded (mutations, snapshots).

Tool Built for Choose Univo when

RapidFuzz Fast similarity between two strings you want duplicate records, not string scores

SemHash Semantic deduplication of text datasets some fields must agree, such as IDs, tenants or environments

Dedupe Matching learned from labelled examples you want results without labelling or training

Splink Probabilistic record linkage at scale, on SQL engines you want an in-process library that also checks records one at a time

Against SemHash 0.5.0, with the same model, on an Apple M2 on mains power:

Univo's search is exact; SemHash's index is approximate. On names alone, neither tool is precise enough to merge automatically, so Univo returns scored candidates, not decisions. Method and every result: benchmarks.

Quickstart · Reference · Benchmarks · Architecture · All docs