返回资讯中心
外部精选
软件工程
#开发者实时流

开发者发布Nenya基础模型:激活参数仅0.5B,训练用60B token

来源标题:0.5B base model trained on 60B tokens rivals Qwen3.5-2B trained on 30T+ tokens

开发者ArsalanZ发布Nenya-2.5B-A0.5B-Base基础模型,采用MoE与混合注意力架构,激活参数仅0.5B,训练数据为60B token。该模型在Hugging Face以Apache-2.0许可开源,作者称其性能可与Qwen3.5-2B相比。模型支持Transformers与vLLM部署。

Hacker News 最新AZDR2026/11/10 · 8 天前30 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

Like 0

Text Generation

Transformers

Safetensors

English

nenya

custom_code

mixture-of-experts

hybrid-attention

base-model

arxiv: 5 papers

License: apache-2.0

Model card Files Files and versions xet

Community

Deploy

Copy to bucket

Use this model Instructions to use ArsalanZ/Nenya-2.5B-A0.5B-Base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

Transformers How to use ArsalanZ/Nenya-2.5B-A0.5B-Base with Transformers:

Use a pipeline as a high-level helper

from transformers import pipeline

pipe = pipeline("text-generation", model="ArsalanZ/Nenya-2.5B-A0.5B-Base", trust_remote_code=True)

pip install -U transformers accelerate

Load model directly

from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("ArsalanZ/Nenya-2.5B-A0.5B-Base", trust_remote_code=True, device_map="auto")

Notebooks

Google Colab

Kaggle

Local Apps Settings

vLLM How to use ArsalanZ/Nenya-2.5B-A0.5B-Base with vLLM:

Install vLLM from pip:

pip install vllm

Start the vLLM server:

vllm serve "ArsalanZ/Nenya-2.5B-A0.5B-Base"

Call the server using curl (OpenAI-compatible API):

curl -X POST "http://localhost:8000/v1/completions"
-H "Content-Type: application/json"
--data '{ "model": "ArsalanZ/Nenya-2.5B-A0.5B-Base", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'

Use Docker

docker model run hf.co/ArsalanZ/Nenya-2.5B-A0.5B-Base

SGLang How to use ArsalanZ/Nenya-2.5B-A0.5B-Base with SGLang:

Install SGLang from pip:

pip install sglang

Start the SGLang server:

python3 -m sglang.launch_server
--model-path "ArsalanZ/Nenya-2.5B-A0.5B-Base"
--host 0.0.0.0
--port 30000

Call the server using curl (OpenAI-compatible API):

curl -X POST "http://localhost:30000/v1/completions"
-H "Content-Type: application/json"
--data '{ "model": "ArsalanZ/Nenya-2.5B-A0.5B-Base", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'

Use Docker images

docker run --gpus all
--shm-size 32g
-p 30000:30000
-v ~/.cache/huggingface:/root/.cache/huggingface
--env "HF_TOKEN=<secret>"
--ipc=host
lmsysorg/sglang:latest
python3 -m sglang.launch_server
--model-path "ArsalanZ/Nenya-2.5B-A0.5B-Base"
--host 0.0.0.0
--port 30000

Call the server using curl (OpenAI-compatible API):

curl -X POST "http://localhost:30000/v1/completions"
-H "Content-Type: application/json"
--data '{ "model": "ArsalanZ/Nenya-2.5B-A0.5B-Base", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'

Docker Model Runner How to use ArsalanZ/Nenya-2.5B-A0.5B-Base with Docker Model Runner:

docker model run hf.co/ArsalanZ/Nenya-2.5B-A0.5B-Base

Model overview Pretraining Benchmarks

UncheatableEval, July 2026 (bits per byte, lower is better)

Code and maths (bits per byte of the reference solution, lower is better)

Verify our results

Quickstart Pretraining Benchmarks

UncheatableEval, July 2026 (bits per byte, lower is better)

Code and maths (bits per byte of the reference solution, lower is better)

Verify our results

Evaluation Pretraining Benchmarks

UncheatableEval, July 2026 (bits per byte, lower is better)

Code and maths (bits per byte of the reference solution, lower is better)

Verify our results

Limitations

License

Citation

Model overview

Nenya is a general purpose 0.5B active, 2.5B total parameter mixture-of-experts (MoE) base model. Nenya rivals Qwen3.5 2B on pretraining benchmarks with 500× lower training compute and 3.4× fewer active FLOPs/token.

Quickstart

Any device with transformers:

import torch from transformers import AutoModelForCausalLM, AutoTokenizer

device = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu" repo = "ArsalanZ/Nenya-2.5B-A0.5B-Base" tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16).to(device)

inputs = tok("The mitochondrion is the organelle responsible for", return_tensors="pt").to(model.device) out = model.generate(**inputs, max_new_tokens=40) # samples: temperature 0.9, top-k 20 (the default) print(tok.decode(out[0], skip_special_tokens=True))

maths, code and short factual answers: greedy decoding

out = model.generate(**inputs, max_new_tokens=40, do_sample=False)

Any device with PyPI

Install it with pip install "nenya[cuda]" on NVIDIA GPUs to get the flash-linear-attention kernels and pip install "nenya" otherwise. With the pip package, trust_remote_code is not needed:

import torch import nenya # registers the model with transformers from transformers import AutoModelForCausalLM, AutoTokenizer

device = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu" repo = "ArsalanZ/Nenya-2.5B-A0.5B-Base" tok = AutoTokenizer.from_pretrained(repo) model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16).to(device)

On a Mac with Apple silicon, use the Metal engine for maximum throughput:

pip install "nenya[mac]"

import nenya print(nenya.generate("The mitochondrion is the organelle responsible for", max_new_tokens=40)) print(nenya.generate("Question: What is the capital of Australia?\nAnswer:", max_new_tokens=8, temperature=0))

It runs on the Mac's GPU, about 125 tok/s on an M4, and downloads its 3.5 GB weight file on first use. It samples with the model's defaults (temperature 0.9, top-k 20); temperature=0 is greedy decoding, for maths, code and short factual answers.

Evaluation

Pretraining Benchmarks

Scored with lm-evaluation-harness 0.4.13 through transformers (--model hf), from log-likelihoods only: no task generates text. Every model runs with the same settings:

Our training data was decontaminated against these benchmarks' evaluation sets.

Four tasks are ours, and their files are in this repo under evaluation/lm_eval_tasks:

Accuracies are in %, ± one standard error.

UncheatableEval, July 2026 (bits per byte, lower is better)

The text was first published 1–25 July 2026 (Jellyfish042/UncheatableEval-2026-07), after every compared model's training data. Each category has 500 documents, each scored as a whole after the model's document-start token, in 4,096-token windows. Each group's average is the plain mean of its categories.

Code and maths (bits per byte of the reference solution, lower is better)

lm-eval scores GSM8K, MBPP and HumanEval only by generating answers. These rows come from the in-repo harness (pretraining/benchmarks.py, context 1,024): the reference models in fp32 with TF32 matmuls, this model in bf16.

Verify our results

The custom task files ship in this repo. On a CUDA GPU:

our runs: torch 2.11.0+cu128 on an RTX PRO 6000 Blackwell, datasets 5.0.1, huggingface_hub 1.33.0

pip install "lm-eval[hf]==0.4.13" "transformers==5.18.0" "torch==2.11.0" "flash-linear-attention==0.5.2"
"datasets==5.0.1" hf download ArsalanZ/Nenya-2.5B-A0.5B-Base --include "evaluation/*" --local-dir nenya_eval TASKS=nenya_eval/evaluation/lm_eval_tasks ARGS=pretrained=ArsalanZ/Nenya-2.5B-A0.5B-Base,trust_remote_code=True,dtype=bfloat16,softmax_dtype=float32,max_length=4096 lm_eval --model hf --model_args $ARGS --include_path $TASKS --output_path results
--batch_size 32 --tasks hellaswag,arc_easy,arc_challenge,winogrande,piqa,sciq,openbookqa,lambada_openai,xstorycloze_en,csqa_cloze lm_eval --model hf --model_args $ARGS --include_path $TASKS --output_path results
--batch_size 16 --tasks mmlu_cloze lm_eval --model hf --model_args $ARGS --include_path $TASKS --output_path results
--batch_size 32 --tasks triviaqa_5shot_greedy lm_eval --model hf --model_args $ARGS --include_path $TASKS --output_path results
--batch_size 2 --tasks uncheatable_2026_07

To score a reference model, set ARGS to its arguments below followed by ,dtype=bfloat16,softmax_dtype=float32,max_length=4096. Qwen3.5, Gemma 4 and SmolLM3 ran at half these batch sizes (their vocabularies are 128k–262k tokens). bf16 scores can move slightly with the GPU and the batch size.

Limitations

A base model. It continues text, it does not follow instructions, and it can produce false, biased or unsafe content.

Trained mostly on English web, educational, academic, code and maths data.

Quality past 16,384 tokens is untested. With no positional encoding there is no hard limit.

License

Apache 2.0: see LICENSE. NOTICE lists the third-party material: nenya_gdn2.py is derived from flash-linear-attention (MIT), and the tokenizer files come from allenai/Olmo-3.1-32B-Instruct (Apache 2.0).

Citation

Nenya builds on Engram, mHC, Gated DeltaNet-2 and Mixture-of-Experts:

% Engram: conditional memory via hashed n-gram lookup @misc{cheng2026conditionalmemoryscalablelookup, title={Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models}, author={Xin Cheng and Rui Tian and Wangding Zeng and Damai Dai and Qinyu Chen and Bingxuan Wang and Zhenda Xie and Kezhao Huang and Xingkai Yu and Chengqi Deng and Shangyan Zhou and Chenggang Zhao and Zhewen Hao and Yukun Li and Han Zhang and Zhengyan Zhang and Yixu Wei and M. Y Xu and Huishuai Zhang and Dongyan Zhao and Wenfeng Liang}, year={2026}, eprint={2601.07372}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2601.07372}, }

% mHC: manifold-constrained hyper-connections @misc{xie2026mhcmanifoldconstrainedhyperconnections, title={mHC: Manifold-Constrained Hyper-Connections}, author={Zhenda Xie and Yixuan Wei and Huanqi Cao and Chenggang Zhao and Chengqi Deng and Jiashi Li and Damai Dai and Huazuo Gao and Jiang Chang and Kuai Yu and Liang Zhao and Shangyan Zhou and Zhean Xu and Zhengyan Zhang and Wangding Zeng and Shengding Hu and Yuqing Wang and Jingyang Yuan and Lean Wang and Wenfeng Liang}, year={2026}, eprint={2512.24880}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2512.24880}, }

% Gated DeltaNet-2 @misc{hatamizadeh2026gateddeltanet2decouplingerase, title={Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention}, author={Ali Hatamizadeh and Yejin Choi and Jan Kautz}, year={2026}, eprint={2605.22791}, archivePrefix={arXiv}, primaryClass={cs.AI}, url={https://arxiv.org/abs/2605.22791}, }

% Mixture-of-Experts: the sparsely-gated MoE layer @misc{shazeer2017outrageouslylargeneuralnetworks, title={Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer}, author={Noam Shazeer and Azalia Mirhoseini and Krzysztof Maziarz and Andy Davis and Quoc Le and Geoffrey Hinton and Jeff Dean}, year={2017}, eprint={1701.06538}, archivePrefix={arXiv}, primaryClass={cs.LG}, url={https://arxiv.org/abs/1701.06538}, }

% Mixture-of-Experts: fine-grained and shared experts (DeepSeekMoE) @misc{dai2024deepseekmoeultimateexpertspecialization, title={DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models}, author={Damai Dai and Chengqi Deng and Chenggang Zhao and R. X. Xu and Huazuo Gao and Deli Chen and Jiashi Li and Wangding Zeng and Xingkai Yu and Y. Wu and Zhenda Xie and Y. K. Li and Panpan Huang and Fuli Luo and Chong Ruan and Zhifang Sui and Wenfeng Liang}, year={2024}, eprint={2401.06066}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2401.06066}, }

Downloads last month 48

","eos_token":"<|endoftext|>","pad_token":"<|pad|>","unk_token":"<|endoftext|>"}},"createdAt":"2026-10-16T20:34:15.000Z","discussionsDisabled":false,"discussionsSorting":"recently-created","downloads":48,"downloadsAllTime":48,"id":"ArsalanZ/Nenya-2.5B-A0.5B-Base","isLikedByUser":false,"availableInferenceProviders":[],"showHuggingChatEntry":false,"inference":"","lastModified":"2026-10-29T17:13:41.00

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →