返回资讯中心
外部精选
软件工程
#开发者实时流

Rewriting model serving around independent workloads and release controls

Push Button Releases: Rewriting Model Serving From the Runtime Up

Hacker News 最新alexanderstatni2026/11/10 · 7 天前22 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

Machine Learning

Platform Engineering

Deep Learning

Artificial Intelligence

Push Button Releases: Rewriting Model Serving From the Runtime Up

Affirm Technology

6 min read·

1 hour ago

--

Author: Tian Luan, Machine Learning & Decision Platform, Affirm

At Affirm, our modeling work spans tree-based models, neural networks, and combinations of the two. Each has to make the journey from a trained artifact to a service that applications can use. Our goal is to make deploying and updating these models routine, even when they need different software or compute resources.

In the serving framework we are replacing, our endpoint tooling generated interface and infrastructure code to deploy each new model endpoint. A single endpoint could generate changes across more than a hundred files. Generation saved manual coding, but model owners still had to review the generated changes and coordinate their deployment. Deployment added another constraint: the shared pipeline deployed the broader fleet even when only one endpoint had changed. Bringing a new endpoint online from scratch could take a couple of weeks.

The framework supported our business for years, but our modeling needs were expanding. For example, GPU workloads required substantial customization because the framework did not support them natively. Supporting a wider range of models meant addressing both the work required to launch an endpoint and the execution environments those endpoints could use.

We considered an incremental approach: dynamic model loading would let the existing service load compatible artifact updates without an application deployment. That would help frequently retrained models, but new endpoints would still require interface and infrastructure changes. It also left further work to support different software and compute requirements.

We needed a common process for introducing models and changing the environments they run in. By reducing the infrastructure work around each release, we aim to shorten the path to production and give modeling teams more time to develop, evaluate, and improve their models.

In Affirm’s next-generation model serving platform, we made the model’s execution environment an independently managed workload. Model owners describe the trained artifact, the runtime image containing the software to run it, and the compute resources it needs. The platform uses that definition to provision or replace the environment without sending unrelated endpoints through a deployment. CPU and GPU workloads can follow the same provisioning and release process.

We divided responsibility across three components:

Figure 1 connects those responsibilities. The control plane uses an infrastructure manager to apply configuration and observe the resulting resources. It publishes endpoint information in a registry that the gateway refreshes. Prediction requests then travel through the gateway and service mesh to the runtime.

Figure 1. Release operations and prediction requests follow separate paths. Dashed arrows show configuration and publication relationships. Responses and operational feedback paths are omitted for clarity.

Each runtime workload specifies its CPU or GPU resources, replica counts or autoscaling configuration, and request batching settings. The platform manages these workloads on Kubernetes. Building on existing infrastructure and serving capabilities lets us focus on model selection, endpoint publication, and release workflows.

Join Medium for free to get updates from this writer.

Our adapters supply the model-specific loading and prediction behavior. Alongside tree and neural models, the runtime supports a fixed hybrid adapter that passes neural-network embeddings to a tree model. These different execution paths sit behind the same gateway and use the same deployment controls.

Consider a model owner introducing a neural network alongside existing CPU models. Assume the network uses an adapter already available in the runtime. Its deployment description includes the following information, shown with conceptual labels rather than API syntax:

Model identity: example-network, version 2 Artifact: stored artifact for version 2 Runtime image: a compatible image, R1 Compute: GPU workers Serving settings: batching and replica configuration

The artifact contains the trained model. The runtime image supplies the software that can load and execute it. Compute and serving settings specify how that software will run. These choices belong together: the same model artifact does not establish which libraries are available, whether a GPU is allocated, or how requests are batched. The actual deployment request also includes schema, placement, and release metadata.

With those inputs, the control plane provisions the workload, evaluates configured readiness conditions, and publishes the endpoint after configured deployment and routing conditions are satisfied. Gateways discover available endpoints through refreshed registry state.

Once version 2 of the neural network is deployed, its owner may want to evaluate it on a selected population before expanding its use. The gateway applies the configured model-selection rules independently of model execution.

A tree model handled by an existing adapter follows the same process with CPU resources. Adding a new framework may require another adapter or a different runtime image, but the surrounding endpoint and release controls remain reusable.

The platform handles provisioning and endpoint publication. Model owners still validate predictions, test load, and decide when to use the endpoint. Application owners must supply the right inputs and connect the caller to an endpoint that exists. This is the practical meaning of “push button”: less repeated infrastructure work between a model and its production use.

Now suppose the neural network needs an updated runtime image, R2. The model remains version 2 while its software environment changes. The service mesh shifts requests between the current and replacement workloads behind the endpoint address. We can introduce the runtime update without redefining which model an experiment assigns to a user.

These are different decisions. A modeling team evaluates whether a new model improves the outcome it cares about. A runtime update needs validation that predictions remain consistent with the existing runtime, alongside checks of latency and error rates. Keeping the controls separate lets teams choose the population for a model experiment independently of the traffic steps used to replace its infrastructure.

For this update, the platform provisions a workload using R2 alongside the one using R1. Keeping the existing workload available during replacement requires temporary capacity for both deployments. The platform shifts traffic in configured steps, provides pause and promotion controls, and waits for a drain period after promotion before removing the old resources. These operations run as durable workflows that retain their progress across interruptions.

Owning that sequence also means accounting for partial completion. Suppose the workload using R2 has been promoted, but deleting the old workload’s resources fails partway through. Some old resources may already be gone. Restarting the traffic shift could send requests toward a workload that can no longer serve them.

The workflow persists sufficient state to distinguish a completed promotion from the remaining cleanup work. If an interruption occurs, a retry resumes from the last confirmed phase rather than replaying an earlier traffic transition.

In our team’s releases during migration from the old platform, deployment execution has taken under 10 minutes in staging and under 15 minutes in production, excluding validation and promotion waits. These early observations cover the deployment operations; the full journey to production also includes evaluation and application integration.

We want to extend the runtime’s existing hybrid support to more general ensemble model graphs, connecting several model computations into one prediction flow. This would let teams compose models through the platform while retaining the common deployment and release controls. General graph execution, resource placement, and validation remain future work.

The test for this platform is whether a team can bring a new model into production with less infrastructure work. A different model may need a new adapter, a different image, or GPU capacity. Those requirements can change while endpoint creation and release management remain shared platform capabilities. That is our ambition: make the last mile from a trained model to production easier to repeat, so teams can spend more of their time improving the models behind our decisions.

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →