Covenant-72B

Like 53

Follow

covenant 56

Text Generation

Safetensors

mlfoundations/dclm-baseline-1.0-parquet

English

llama

arxiv: 2603.08163

License: apache-2.0

Model card Files Files and versions xet

Community 1

Copy to bucket

Covenant-72B Model Overview

Usage

Model Details

Technical Specifications

Performance on Benchmarks

Covenant-72B

Model Overview

Covenant-72B is the largest permissionless collaboratively trained language model, trained entirely from scratch at the 72 billion parameter scale on 1.1 trillion tokens of English text.

For more details, see the technical report. This is a base model. See Covenant-72B-Chat for the instruction-tuned variant.

Covenant-72B was trained with 20+ globally distributed participants coordinated via decentralized infrastructure on the Bittensor blockchain. Unlike prior collaborative training efforts that use whitelisted compute, Covenant-72B is the first to achieve this scale with fully permissionless participation. Training used the SparseLoCo communication-efficient optimizer to reduce bandwidth requirements across distributed nodes.

Usage

import torch from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained( "1Covenant/Covenant-72B", torch_dtype=torch.bfloat16, device_map="auto", ) tokenizer = AutoTokenizer.from_pretrained("1Covenant/Covenant-72B")

input_text = "The theory of general relativity" input_ids = tokenizer.encode(input_text, return_tensors="pt").to(model.device) output_ids = model.generate(input_ids, max_new_tokens=100) print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

Model Details

Compute Participants: 20+ independent contributors on Bittensor

Minimum Compute per Participant: 8×B200 or equivalent

Model License: Apache 2.0

Technical Specifications

Parameter Value

Parameter Size 72B

Architecture LLaMA-style (LlamaForCausalLM)

Number of Layers 80

Number of Attention Heads 64 (8 KV heads)

Hidden Size 8192

Intermediate Size 28672

Head Dimension 128

Vocabulary Size 262,144

Training Details:

Performance on Benchmarks

All results are 0-shot acc_norm (%) unless noted.

","eos_token":"<eos>","pad_token":"<eos>","unk_token":"<unk>","use_default_system_prompt":false}},"createdAt":"2025-10-01T16:03:44.000Z","discussionsDisabled":false,"discussionsSorting":"recently-created","downloads":81,"downloadsAllTime":1358,"id":"1Covenant/Covenant-72B","isLikedByUser":false,"availableInferenceProviders":[],"showHuggingChatEntry":false,"inference":"","lastModified":"2026-03-10T18:50:26.000Z","likes":53,"pipeline_tag":"text-generation","librariesOther":[],"trackDownloads":true,"model-index":null,"private":false,"repoType":"model","gated":false,"tags":["safetensors","llama","text-generation","en","dataset:mlfoundations/dclm-baseline-1.0-parquet","arxiv:2603.08163","license:apache-2.0","region:us"],"tag_objs":[{"id":"text-generation","label":"Text Generation","type":"pipeline_tag","subType":"nlp"},{"id":"safetensors","label":"Safetensors","type":"library"},{"id":"dataset:mlfoundations/dclm-baseline-1.0-parquet","label":"mlfoundations/dclm-baseline-1.0-parquet","type":"dataset","extra":{"disabled":false}},{"id":"en","label":"English","type":"language"},{"id":"llama","label":"llama","type":"other","clickable":true},{"id":"arxiv:2603.08163","label":"arxiv:2603.08163","type":"arxiv","extra":{"paperTitle":"Covenant-72B: Pre-Training a 72B LLM with Trustless Peers Over-the-Internet"}},{"id":"license:apache-2.0","label":"apache-2.0","type":"license"},{"type":"region","label":"🇺🇸 Region: US","id":"region:us"}],"widgetData":[{"text":"Hi, what can you help me with?"},{"text":"What is 84 * 3 / 2?"},{"text":"Tell me an interesting fact about the universe!"},{"text":"Explain quantum computing in simple terms."}],"safetensors":{"parameters":{"F32":72747327488},"total":72747327488,"sharded":true,"totalFileSize":290989454372},"hasBlockedOids":false,"region":"us","isQuantized":false},"inferenceContextData":{"billableEntities":[],"entityName2Providers":{}},"canWrite":false}">Safetensors

Model size

73B params

Tensor type

F32 ·

Files info