Local Inference with Large Language Models

To head off any criticism: not a word of this text was written by AI, all the mistakes are my own. If you should spot any I’d be very happy if you let me know.

This article assume basic familiarity with the terminology around LLMs and the various moving parts, I’ve tried to explain as much as possible but in case something is unclear feel free to reach out and I’ll expand the article or clarify in an email.

Why make a set to run an LLM locally if the bleeding edge is affordable and easily accessible?

I’ve been playing with LLMs since the first open models were released and even though at the time the results weren’t all that impressive it was clear that it wasn’t going to stop there. Over the last couple of years the model capabilities have improved at a breakneck pace and today these models are extremely capable when used appropriately. The business that I run sells a service to investors and companies that are looking to be acquired that is highly confidential in nature. Effectively we ask other companies to share pretty much everything, both the good and the bad with our team so that we can come to a balanced verdict in our report. To enable this sharing we sign NDAs (non-disclosure-agreements) that usually have scary numbers in them in case we would breach that confidentiality, and rightly so, this kind of data is easily weaponized.

As part of our process we write reports, our ‘work product’ and these reports need to be proofread. The problem is that typically these reports are due on Friday evening and our draft version is usually only ready somewhere between 2 and 3pm on Friday afternoon. That leaves very little time for proofreading, usually we try to get one or two passes in before the document is handed to a customer but it isn’t rare at all that on second or third reading during the weekend we still find mistakes. These can be small and innocent or the could be larger (cf: typos and grammar errors or errors in the content itself). Given infinite time I’m sure we would find all of these but we simply don’t have that luxury.

Now, the easy way out would be to upload the document just prior to sending it out to an AI service provider, have that service do the proofreading and then we would apply the corrections (there is absolutely no way that I would allow an AI control or direct input into our work process). But that would cause us to break our NDA. And that’s before we get into the hairy details of questions such as ‘Can we trust Altman or Amodei when they say they won’t use the data for their training?’, ‘Even if we can trust them, can we trust everybody who works there?’ and ‘And even if we can, are we sure that shipping these files out at all is a wise decision?’. The answer the last one was the easiest, that was simply ‘No.’. So we decided to bite the bullet and do this locally.

We have some other reasons as well: we don’t particularly like the idea of a large entity vacuuming up the whole of human culture and then selling it back to us, no matter how nicely packaged and how useful it is. That feels like a very unethical thing to do and the providers of ‘open weights’ models are closer to how we think this should be handled. Second to that is that we don’t want to end up in a situation where an outside party has control over our workflow and can tell us what we are - or are not - allowed to do. ‘Sorry Jacques, I can’t let you do that’ is not something that I would ever accept from a computer, computers do what I tell them to do, not what they allow.

These locally run models are not precisely as good as the big frontier models. They tend to be a bit smaller, sometimes a bit older and probably not quite as clever. But depending on your workload and other demands (such as confidentiality) they can be a surprisingly good match. And in case the likes of OpenAI decide to hack your server they may well be your only means of defense.

If you want to do inference locally you’re going to need a couple of things:

There are a small number of major engines suitable for deployment on your own hardware, a much larger number of derivatives (sometimes weakly derived, sometimes not much more than a relabeling) and then there are numerous smaller efforts. We only considered open source engines, and more specifically engines that we have actually used (benchmarked, and used in production). Let’s focus on the main engines first. These are the ones that we used that are either the ‘root’ projects or independent projects that have shown sufficient staying power that you could presumably build a business on them:

Llama.cpp (WP) is probably one of the easiest ways to get started with local llms. The project is now 3 1⁄2 years old, stable and extremely versatile. Installing it on a wide variety of hardware is usually a matter of cloning the github repository and a single command to configure and another to build it. It’s not necessarily the fastest inference engine but it is super reliable and the quality of the code is quite high. Developed initially by Georgi Gerganov it is now managed by a team of seasoned developers managed by Georgi, who still contributes quite a bit to the code. We have used llama.cpp on pretty much all of our hardware and with a really large variety of models. Occasionally you’ll run into a lack of support for a model that has just been released but usually within weeks (sometimes days) these things are resolved. Llama.cpp is not usually the inference engine used by commercial entities, where the last little bit of performance weighs heavier than the convenience factor that Llama.cpp brings to the table. The project is named after the ‘Meta’ (ok, Facebook) model.

Good guides exist on how to get it working, the file format (.gguf) is widespread and conversion utilities are available in case a model is not shipped in a format that Llama.cpp can use. There are also groups specializing in converting and quantizing models to make it fit on particular hardware. Updates are a daily affair, it is usually a good idea to keep an eye on the GitHub page to see what they’re up to and whether or not it may be useful to you.

VLLM is a powerful inference engine that is far more complex than most other engines. The downside of all this complexity is that installing, configurating and running it can be a significant task. The upside is that it tends to extract more performance from the same hardware than other engines, if that hardware is a good match for VLLM. If you intend to use VLLM you need to ensure that the hardware is a good match, usually by purchasing the hardware with the specific goal of running VLLM. Manufacturers of AI specific hardware may choose to support the ‘recipe’ for their hardware to ensure that VLLM runs well. This can be a shortcut to a powerful solution. But figuring this all out on your own can be extremely time consuming. VLLM is the inference engine of choice for many commercial LLM service providers. For them it is simply a matter of money, it is cheaper to invest the up-front effort and then to reap the benefits from a (usually marginal, but there are exceptions) system that performs better.

vLLM setups tend to be a bit more fragile, require more expertise and are less likely to function well ‘out of the box’. But if are willing to spend the time, and already have some expertise this may well be a good match for you. VLLM configurations typically use the safetensor format that most models ship in from the model provider, so normally speaking no conversion step is necessary.

SGLang sits somewhere in the middle between vLLM and Llama.cpp when it comes to ease of installation and ease of use. It usually squeezes a bit more performance out of the hardware than Llama.cpp but this comes at a price, we found that for our workloads on a pretty varied bunch of CPUs we did not see enough gain in performance to warrant further pursuit, but it is possible that we simply did not spend enough time to really unlock this engine. Installing it is a matter of running a docker container. This makes auditing what you are running a bit harder than with the other two packages (and where Llama.cpp wins hands down because of its simplicity).

DwarfStar is an inference engine produced by Salvatore Sanfilippo of ‘Redis’ fame. Salvatore is an extremely accomplished and productive programmer and DwarfStar is an interesting variation on the inference theme. Rather than to make an inference engine for a hundred different models and 500 different hardware combinations he focused on (initially) a single model (specifically DeepSeek, aka DS which is probably where the project got its name) and a Mac with unified memory. Since the launch many more platforms are supported as well as several more models. DwarfStar is easy to use and easy to configure, it is regularly updated with new features and/or performance improvements.

Ninfer is, like DwarfStar, a specialized engine. Initially made to run Qwen 3.5 and since adapted to be able to run other models as well it squeezes ever last drop of performance out of a single NVidia 5000 series GPUs. If your setup matches that description then this is most likely the best engine for you assuming your model is available as a ‘.ninfer’ artifact (beware, there are multiple versions of these that are not backwards compatible so you’ll have to convert). Ninfer is in active maintenance and, interestingly, there are already many forks for hardware that is similar but not identical to the hardware that ninfer was originally developed on. For instance, there are forks for the 3000 series GPUs (which includes the venerable 3090, which is still a powerhouse, espcially given its age) and the 4000 series as well.

Well, it does and it doesn’t. As long as the engine exposes an OpenAI compatible endpoint you don’t neccesarily care all that much about which engine you are running, as long as (1) it runs well, (2) performance is acceptable and (3) your model of choice is supported.

There are thousands (if not 10’s of thousands) of different models. These range from general purpose models targeted to specific sizes of VRAM or some other inference time limitation to models tuned for a particular application. The main factors distinguishing the models from each other are:

We could not possibly list them all, but at the time of this writing there are a couple of these that deserve special mention here:

The Qwen family of models ( is interesting because they span a range of capabilities and sizes, from small (7 GB or so, small enough for even the smaller video cards) all the way to 400GB and by virtue of quantization many different sizes in between. One of our workhorse models is Qwen 3.8 27B, we use this model for coding and robotics experiments (not the original application of our AI rig, but now that we have it, why not…).

DeepSeek is a very impressive model, in fact, it was the first release of DeepSeek that prompted us to make the next step with this project because we considered the quality ‘good enough’ for our usecase. It is not necessarily the best model overall but it strikes a good balance between speed and size and is quite versatile.

GLM, in particular GLM 5.3 is an extremely capable model. It is also the largest model that we can still comfortably run on our hardware. Due to its size it is not super fast but it more than makes up for that in capability, so much so that it can perform better wall-clock time wise than other smaller models that will use many more tokens and/or many more turns to get the same job done. GLM 5.3 is also very good at cyber related tasks. To this day it has never refused an assignment.

This