Show HN: I trained a 160M code model on one GPU and got 22.56% on HumanEval

EUGEN 160M — my code model, tap to try it

EUGEN 160M

EUGEN 160M is a small language model that helps with writing code: it completes code as you type (autocomplete) and suggests the next line. It is also trained to fill in code in the middle of a file (fill-in-the-middle) and specialises in translating Python into C++. It has 164.5 million parameters and is trained from scratch on a single NVIDIA RTX PRO 6000 (96 GB) GPU. Its small size makes generation fast on an ordinary CPU as well, at up to 364 tokens per second. You can try the model in the terminal: just click the button below.

Try EUGEN 160MYou should know that…Model results

Training is at an early stage, and the results already give reason for optimism. On HumanEval, where the model writes a Python function from its description, EUGEN's best measured results are 22.6% of tasks with one attempt and 37.2% with ten; in recent measurements the ten-attempt result is about 34%. In verified Python-to-C++ translation – the compiler and tests pick a working version from several of the model's proposals – 77 of 100 programs compile and 53 pass all tests. The model's stack makes sure that only translation proposals which passed these checks reach the user. The chart sets EUGEN's results against models of up to 0.5 billion parameters.

Compared with other modelsModel training reportWhat I learned from training my own model

EUGEN is a decoder-only transformer with 36 layers, trained with MuonH, a variant of Muon that keeps the norm of the weight matrices fixed. The model is in base training. About 0.35% of the data are our own: STaR solutions generated by EUGEN itself that passed their tests, and translation pairs between Python and C++, Java, Go and Rust, checked by compilation and tests.

I also work on optimising and speeding up model training with a sparse FFN, and for this I developed my own GPU kernel, written in Triton. In a model of 306 million parameters (FFN 4,096), the optimised kernel reached 36% higher throughput than its first version. Our early tests show that with a larger model and an FFN of 8,192, the sparse kernel from the EUGEN project can deliver a speed-up of more than 40% over a dense model. Whether a model trained sparse and densified at the end matches a dense model remains open, but the early results lead me to train the next, larger model this way.

TetherGate is a mechanism that keeps learned gates from closing. A gate scaling a residual branch – an expert, an adapter or a newly added layer – tends to be pushed gradually to zero by the optimiser, after which the branch contributes nothing to the model. TetherGate combines a floor driven by the training step with a spring that pulls the gate's threshold towards a moving anchor computed from the current control signal. The signal can be any statistic. I have tested the mechanism experimentally in my own models.

Download TetherGate (.zip)TetherGate on Hugging Facetethergate.py

I keep doing my own research into artificial intelligence. I want to know whether we can find signals inside a model that will help us detect hallucinations or bad intentions. To that end, I study the activation signals of the model's internal layers, looking for indicators that would help predict and monitor such behaviour. I have no formal education in the field, but I aim to work as professionally as possible: with publicly released data, repeatable measurements and clearly stated limitations.

A few thoughts on AI safetyResearch papers

I don't write code by hand. I design the tasks and run teams of AI agents, and I check every result. A team works in three stages: research, execution and verification. Execution is overseen by two independent controllers: one keeps to the scope of the task, the other to its technical quality.

I run two or three such teams in parallel, so when one task needs my attention, the others keep going. I delegate large chunks of work, but only up to the point where I can still check them properly. Whether something is done is decided by tests or measurements, not by the model's claim.

Thoughts on AI safetyCoherence Over TruthHDK: LLM audit trailsStudy: before the model answersCorrection of my earlier papers

View full file

source —

how-i-started.md

You should know that…

A reading game: while you read steadily, a robot eats a line of this site’s code and scores points; scrolling too fast or unevenly makes it miss characters. Decorative — it never affects the text.

…I have no formal education in programming or in training LLMs.

I got into building apps and training AI through a thoroughly strange chain of coincidences which, looking back, seems to make quite a lot of sense. And I think it says something important about our times. I hadn’t planned it; I never imagined I’d have more to do with technology than the average iPhone user. For years my only window onto the world of Big Tech was the Hard Fork podcast. Shout-out to Casey and Kevin :)

It all started with vibe coding and the OrsonAI app. Playing around with AI, I began building tools that could help me in my day-to-day work as a journalist. Transcribing interviews while they were being recorded and importing them into an editor, or an editing mode that let me consult a chatbot about selected passages without switching windows and pasting in the whole context — at the time these were genuinely neat tools, and they sped my work up considerably.

Over time I began to think of it as a project, something that could serve other people too. The idea was simple: take the features that save me time and build a product for people facing the same problems. I spent a lot of time building an interface and mechanisms that could deliver safe AI for newsrooms — AI that helps the journalist but doesn’t do the job for them. I even designed a system that issues certificates of human authorship for a text, published on the Hedera Hashgraph public ledger. An immutable record with evidentiary value in court, showing how the text came into being and what part a human played in writing it.

Here you will find a description of the HDK technologyHere you can download the free HDK libraryHDK on Hugging Face

But I was in for a big disappointment. The reception from the industry people I showed the app to was frosty, to say the least. The question that came up at the end of every presentation made me painfully aware of how wrong I’d been: “Why would anyone pay for your app when we’ve got ChatGPT for free?” Things like a certificate of human authorship, data protection or even a substantial improvement in workflow over what an ordinary chatbot offered didn’t matter. I heard this from people who, day in, day out, stress the importance of reliable journalism.

Still, working on the app, and on an agentic AI system reliable enough to be used in newsrooms, pushed me in a completely new direction. Trying to understand better where language-model hallucinations come from, I started reading research papers on arXiv, and eventually designed and ran a series of studies on SwiGLU gate activations in several open language models. I wrote up the findings in a few papers that I posted on ResearchGate. Looking back, some of the claims in those papers strike me as megalomaniacal, and the writing itself would probably need revising and a more scientific vocabulary, but I’m not taking them down — if only for the sake of research integrity.

Here you will find my papers on the subject

Today, after many failed experiments and disappointments, and with more humility earned on the proving ground of language-model training, I went back over those results myself. Some of what I had taken for a discovery turned out to be a simpler effect, so I replaced the old papers with a single, corrected one. The question everything started with is still the same, though: can we find signals inside a model that help us detect hallucinations or bad intentions? That is what my research is about to this day.

The work consumed me completely. Perhaps that’s somewhat in keeping with my character: I like to throw myself into things that are a challenge but also let me learn something about the world around me and understand it better. Besides, before I went into journalism I studied philosophy, with a particular interest in the philosophy of science and epistemology.

Here you will find an essay summing up some of my thoughts

Training my own model ended in total disaster. I probably made every mistake in the book, despite my habit of thorough research and the vast amount of specialist literature I’d devoured. The first results looked promising, but only later did I understand that they were largely down to noise regularisation. The model’s advantage in ablation tests didn’t justify the extra compute needed to calculate the Gini of the gate activations. On top of that came purely technical training challenges. When my modifications were exempted from normalisation, they held back effective learning, whereas normalising them flattened them out, turning the model into a standard transformer that was just more expensive to train. I’m making the architecture details available for download here — perhaps someone will find useful material in it for learning from other people’s mistakes.

One solution that I think still deserves attention is the M2 gate. Thanks to an affine anchor, it can keep running on a sigmoid without the risk of being dragged down by a lazy optimiser looking for an easy way to lower the training loss. In my case this gate, trained on the Gini signals of the activations, was meant to cut noise during the model’s generation. However, taking the signal out from under normalisation — as I mentioned earlier — led to a model with an excessively peaked confidence distribution: logically sharper, but in practice worse at learning facts. Perhaps this solution could be useful in other models, for gating other components, and help the training process.

In the end I came to see all my failures in training a model on my own architecture as capital that shouldn’t be wasted. Today I’m training my own small model, EUGEN 160M, which specialises in code. Comparing its benchmark results so far, I’m cautiously optimistic and expect this won’t be time wasted. I decided to train a model specialising in backend languages because with this kind of architecture it’s easier to monitor progress, and the model itself can be a useful autocomplete tool for inference on an ordinary CPU. You can try the model in a specially built terminal on my site and compare it with other open models of a similar size.

The results so far are above all down to rigorously tracking the model’s progress and the mistakes it makes during generation. That let me intervene on the fly, adding training data to strengthen the model wherever it showed weaknesses. I also regularly used the STaR and LEMA techniques, creating synthetic data from what the model itself had generated, as well as translation data that teaches the model the logic of code by showing it examples of translations between Python and C++, Java and Rust. All this material currently makes up about 0.35% of the whole dataset, and its share will be raised gradually as training goes on.

Training the model is great fun and — like journalism before it — gives me the sense that I’m doing something important: something that lets me better grasp the changes in the world around me. I can’t write complex C++ from scratch on my own, that’s a fact, but I try to understand the logic of the processes. That’s how I chalk up the small wins that let me keep moving forward in my work. And yes — with the help of an AI coding agent I built my own Triton kernel that makes it possible to train a sparse model, improving TPS (Tokens Per Second) by 23% over a dense model; later kernel optimisations added a further 36%. I intend to use it to train my next, somewhat larger model.

You can download the kernel hereKernel on Hugging Face

And coming b