
Running Apertus: A Swiss-Made Multimodal LLM
Nowadays in enterprise software engineering there are two primary models of using LLMs and provision them to a fleet of users:
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
x504
Oct 20, 2026
Share
Nowadays in enterprise software engineering there are two primary models of using LLMs and provision them to a fleet of users:
Commercial Public APIs: Using closed-source models, such as Claude Opus , either directly through vendors like Anthropic or via cloud providers offering managed access to these models, such as Amazon Bedrock and Microsoft Azure AI Foundry.
One of the downsides is privacy as companies are sending private information and proprietary code.
Other problem the developers are facing is deprecation. The vendors tends to deprecate old models. Such models are used by applications in production that were carefully instrumented and designed to enhance the integration with these models. Forcing developers to go through a complete evaluation, benchmark and Blue/Green deployment process before the model is complete deprecated.
There are more concerns. For instance, today it remains unclear whether the judgments done by LLMs are implicitly shaped by geopolitical cues.
Image found in the internet - October 2026
On-Premise Clusters: Deploying open-weights models, such as DeepSeek, Qwen, AMALIA (A Multimodal focused in Portuguese languague and the Portuguese culture), APERTUS (Fully Open Foundation Model for Sovereign AI developed in Switzerland) on internal hardware, such as NVIDIA. These deployments often involved aggressive quantization techniques (e.g., NVFP4) to optimize the underlaying hardware achieving higher throughput and lower latency compared with the base model while keeping all data interaction within the corporate network.
There are of course challenges when running self-hosted LLMs. Few of them are:
Computing. You have to own your own GPUs.
Knowledge. You have to know how to run an inference-distributed cluster and maintain it.
Operability. As everything you own, you have to maintain, monitor and fix when problems appears
I strongly believe we should reduce our dependency on external LLM vendors. They could change their terms, pricing, or policies at any time, potentially putting our workloads and data at risk. We need to achieve AI and LLM sovereignty by maintaining control over our infrastructure, workloads, and data. The internal data is one of the company’s most valuable assets, and we must ensure it remains private and under our control.
In this post, I’ll show you how easy it is to run APERTUS on your own server using the vLLM framework.
Before running anything or make any hardware decisions, we need to understand the task at hand. This starts with getting familiar with the characteristics of the LLM we want to run.
We are going to run the 8B (8-billion-parameter) version of APERTUS. Given that the model uses the BF16 (bfloat16) data type, we will need at least 32 GB of GPU VRAM to run it comfortably.
This is a multilingual model optimized for smaller GPUs. You can learn more about it here.
In order to enable thinking mode, we would need to set --reasoning-parser and --default-chat-template-kwargs.enable_thinking .
As today (October 2026) the Apertus 1.5 release has not been completely merged in the Transformers and vLLM libraries hence we will need to use at least --tool-call-parser as extra Argument when running vLLM.
The accelerator we are going to use is 1 x Nvidia RTX A6000 with 48GB VRAM. Which is enough to run the model we describe above for our own purposes.
The framework we are going to use to run inference is the very well stablished vLLM.
vLLM comes with a lot of built-in ready optimizations that allow run faster inference servers. Some of its features are:
Efficient management of attention key and value memory with PagedAttention.
Continuous batching of incoming requests, chunked prefill, prefix caching
We are running on an Ubuntu Server 24.04 LTS. Let’s install few pre-requisites packages. For the sake of this post we are going to use the vLLM Production-stack which is a reference stack for production deployments.
Initial OS packages updates
apt-get update
Clone production-stack repository
git clone https://github.com/vllm-project/production-stack.git cd production-stack/utils
install kubectl and verify it is installed
bash install-kubectl.sh
kubectl version --client
Expected similar output:
Client Version: v1.37.1 Kustomize Version: v5.8.1
Install helm and verify it is installed
bash install-helm.sh
helm version
Expected similar output:
version.BuildInfo{Version:"v3.22.1", GitCommit:"d6c1a0e4027caa6abc7892e7ea9f0cda7447baa3", GitTreeState:"clean", GoVersion:"go1.26.9"}
Instal helm diff plugin
Source: https://github.com/databus23/helm-diff
helm plugin install https://github.com/databus23/helm-diff
Expected similar output:
Installed plugin: diff
Installing Minikube with GPU support
sudo apt remove minikube
bash install-minikube-cluster.sh
Verifying GPU Configuration
minikube status
Expected similar output:
minikube type: Control Plane host: Running kubelet: Running apiserver: Running kubeconfig: Configured
kubectl describe nodes | grep -i gpu
Expected similar output:
nvidia.com/gpu: 1
Run a test pod in kubernetes
kubectl run gpu-test --image=nvidia/cuda:12.2.0-runtime-ubuntu22.04 --restart=Never -- nvidia-smi
Wait for the pod to be in phase: completed, then:
kubectl logs gpu-test
Expected similar output:
+-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 570.195.03 Driver Version: 570.195.03 CUDA Version: 12.8 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA RTX A6000 On | 00000000:00:06.0 Off | Off | | 30% 34C P8 35W / 300W | 1MiB / 49140MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+
Easy. At this point we have a Kubernetes cluster using minikube with the Nvidia plugin installed in the cluster and ready to run any GPU workload. Let’s now deploy APERTUS with vLLM.
Deploy vLLM Instance (model: vllm_apertus_1.5_release)
helm repo add vllm https://vllm-project.github.io/production-stack
Create a file apertus.yaml with the following (or similar) content
DO NOT FORGET to set your Hugging Face token in 'hf_token'
servingEngineSpec: runtimeClassName: "" modelSpec:
- name: "apertus-8b" repository: "ghcr.io/swiss-ai/vllm_apertus_1.5_release" tag: "latest-amd64" modelURL: "swiss-ai/Apertus-v1.5-8B" replicaCount: 1
requestCPU: 10 requestMemory: "16Gi" requestGPU: 1
pvcStorage: "50Gi" pvcAccessMode:
- ReadWriteOnce
vllmConfig: enableChunkedPrefill: true enablePrefixCaching: true maxModelLen: 16384 extraArgs: ["--gpu-memory-utilization", "0.9", "--enable-auto-tool-choice", "--tool-call-parser", "apertus"]
hf_token: hf_1234..
See first what is going to be installed
helm diff upgrade --install apertus vllm/vllm-stack -f apertus.yaml
Then deploy
helm upgrade --install apertus vllm/vllm-stack -f apertus.yaml
Confirm the helm chart was installed
helm list NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION apertus default 1 2026-10-19 19:34:35.173857134 +0000 UTC deployed vllm-stack-0.1.13
List the pods an you should see two pods being created
kubectl get pods NAME READY STATUS RESTARTS AGE apertus-apertus-8b-deployment-vllm-5c6f6bf484-7twjx 0/1 ContainerCreating 0 26s apertus-deployment-router-76c84b85fb-c49sx 0/1 ContainerCreating 0 26s
You could inspect the pod: apertus-8b-deployment-vllm and wills ee it is fetching the docker image
which contains the model weights. it will take few minutes.
spec.containers{vllm}: Pulling image "ghcr.io/swiss-ai/vllm_apertus_1.5_release:latest-amd64"
We need to wait until the model is pulled from HuggingFace.
Then the apertus deployment pod is going to change phase to running. I took few screenshots to give you an idea of the startup process. This will take few minutes to the engine to load all model weights into GPU and start the engine server.
Starting vLLM server on http://0.0.0.0:8000 Available routes are: Route: /openapi.json, Methods: GET, HEAD Route: /docs, Methods: GET, HEAD Route: /docs/oauth2-redirect, Methods: GET, HEAD ...
Let’s confirm our inference server is working properly and ready to take prompts. An easy way is to expose the kubernetes service to the localhost of the Ubuntu server machine and then in another terminal run a the curl command below
Expose vLLM server to port 30080 on localhost
kubectl port-forward svc/apertus-router-service 30080:80
Then get the models available on our vLLM server
curl -o- http://localhost:30080/v1/models
You should get a response similar to this:
You can see model swiss-ai/Apertus-v1.5-8B is available! 🎉
Let’s now instal a tiny LLM CLI tool which comes handy to use for sending prompts and streaming the output back to the user in the terminal.
This is not the final interface you would like to give to users. This is only for educational purposes.
As a final interface to users you can use any of these open source solutions:
As a second test let’s open 6 terminals and send the following prompt to see our LLM working!.
Prompt: “Describe kubernetes in 200000 words“
We can confirm our vLLM deployment is working and ready to serve to users. Let’s now use few benchmarking tools to figure out the limits of our deployment (Model-Hardware-Workload)
For educational purposes we are going to start with vLLM bench. This is an “old“ way provided by vLLm team to benchmark inference systems. it is written in Rust and is basically a client sending different and multiple prompts of different input-len at different rates to your inference server.
In order to take advantage that vLLM libraries are already installed in the deployment Kubernetes pod, we can run vllm-bench inside the pod.
List the pods
kubectl get pods
Exec into the apertus deployment pod
kubectl exec -it apertus-apertus-8b-deployment-vllm-6df55cbc8-gpcgl -- /bin/bash
Run a benchmark with the following configuration
vllm bench serve
--backend openai
--base-url http://localhost:8000
--endpoint /v1/completions
--model swiss-ai/Apertus-v1.5-8B
--dataset-name random
--num-prompts 500
--random-input-len 512
--random-output-len 1024
--request-rate 5
--max-concurrency 32
--temperature=0
At the end of the run you will get a report in the terminal. It could as well generate a JSON report for generating graphs afterwards. The screenshot below shows you while running the benchmark.
These metrics are very important, specially TTFT, ITL, TPOT and Request throughput.
Let’s run it again changing input-len from 512 to 1024. We expect the ITL, TPOT won’t change but the TTFT should increase as we are sending a longest prompt. We can confirm this in the results:
Now let’s change the request rate from 5 to 15 and max concurrency from 32 to 64. Now we expect ITL and TPOT be affected as we will be more close to compute-bound.
Here I use a basic configuration for benchmarking the LLM but you could use different datasets, even a custom one and way more advanced configurations.
Let’s now use a more modern benchmarking tool and which is the one recommended as today by the vLLM maintainers.
GuideLLM allows you to evaluate how LLMs perform under real workloads and configurations. It simulates end-to-end i
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。