Volcano v1.16 Released: NamespaceQueue, fairshare, and a broader HyperNode ecosystem

Volcano v1.16.0 is available . v1.16.0 introduces NamespaceQueue and the fairshare plugin. Tenants can use hierarchical queues within their own namespaces to organize quotas and queue hierarchies within administrator-authorized resource pools. Fairshare brings historical resource consumption into scheduling decisions among namespaces sharing a queue, improving fairness across teams. This release also expands the HyperNode ecosystem. Platforms using other schedulers can deploy the HyperNode controller on its own to discover topology and manage supernodes and their hierarchy, then integrate that information through the HyperNode API. For inference services using Volcano, new PodGroup topology anti-affinity rules can spread instances across supernodes to reduce exposure to a shared failure domain. Scheduler performance improvements include an Unschedulable Job Cache that records why Jobs could not be scheduled and skips repeated checks until blocking conditions change. Node scoring, cache updates, and queue lookups also become more efficient. The release additionally brings Kubernetes 1.36 support, load estimation for bulk Pod dispatch, DRA scoring, and accelerator and stability improvements.

Highlights

• Scheduler Performance and Scalability: An unschedulable Job cache reduces backlog retries, alongside improvements to node scoring, cache updates, and queue lookups.
• Namespace-local Queue Hierarchies: Tenants can manage hierarchical queues, quotas, and resource sharing within their namespaces and administrator-authorized resource pools.
• Long-Term Fairness with fairshare: Slurm-inspired usage decay improves fairness for tenants isolated by namespace and sharing a queue.
• PodGroup Topology Anti-Affinity: Required and preferred rules separate workload instances across HyperNode domains.
• A Broader HyperNode Ecosystem: Other schedulers can reuse topology discovery and supernode management without running the other Volcano components.

Key Features

1. Scheduler performance and scalability

v1.16.0 introduces the Unschedulable Job Cache to improve scheduling under large pending backlogs, alongside improvements to node scoring, cache updates, and queue lookups. Unschedulable Job Cache A Job may remain Pending because its queue is full, the cluster lacks suitable resources, or no node satisfies its placement constraints. Repeating the same scheduling checks in every session adds work without changing that outcome. With a large backlog, these retries also compete with newly submitted Jobs that could run. The new UnschedulableJobCache records why a Job could not be scheduled. Plugins provide event hints that identify changes which could make the Job schedulable. Subsequent sessions skip the checks covered by the cached result until a relevant change requests a retry. This reduces redundant work and improves scheduling performance in clusters with persistent unschedulable backlogs. Key capabilities:

• Selective skipping: Uses rejection causes to skip enqueue checks, whole-Job evaluation, or checks for particular tasks while respecting gang requirements.
• Event-driven wakeups: A Node label update can wake a Job blocked by node affinity. Resource release or a Queue change can wake Jobs blocked by resources or quota. Storage and DRA events are also supported through plugin hints.
• Safe fallback: A Job is cached only when every rejecting plugin provides a safe hint. Other Jobs continue through normal per-session evaluation.
• Preserved scheduling semantics: Cached Jobs remain visible to pending demand, queue accounting, and gang requirements. Normal checks run again before placement after a wakeup.
• Bounded retries: A watchdog retries cached Jobs after a configurable maximum skip duration, defaulting to five minutes. The in-memory cache is cleared on scheduler restart. The feature is disabled by default. Enable it with the scheduler feature gate:

helm upgrade --install volcano volcano-sh/volcano \
  --namespace volcano-system --create-namespace --version 1.16.0 \
  --set custom.scheduler_feature_gates="UnschedulableJobCache=true"

Initial coverage includes the upstream predicates integrated by Volcano, native Resource Fit, proportion, and non-hierarchical capacity. Hierarchical capacity rejections and rejections from plugins without hint providers continue to be evaluated each session. Optional per-Job debug metrics report skips, wakeups, and watchdog expirations. Related:

• PRs: #5848, #5644
• Design and performance results: Unschedulable Job Cache
• User guide: Enable the Unschedulable Job Cache
• Contributor: @JesseStutler Multiple scoring plugins can now score nodes in a shared traversal, reducing scoring overhead when several plugins are active. Node updates and queue lookups also avoid unnecessary object copying and traversal. (#5928, #5556, #5506) Volcano will continue to improve performance and scalability. Future work will explore scheduling efficiency, cache and event processing, and status updates, guided by workload benchmarks and production feedback.

2. NamespaceQueue brings queue hierarchies to Namespaces

Platforms use different tenancy models. Some assign resource pools to tenants through cluster-scoped Queues. Others use namespaces as the management boundary for teams or projects and want tenants to configure queue quotas and hierarchies within that boundary. NamespaceQueue supports this namespace-based model, aligning queue management with tenant ownership. NamespaceQueue introduces a namespaced resource with the same resource-governance concepts as Queue, including capability, deserved, guarantee, and reclaimable. Administrators authorize a namespace to attach its queue hierarchy to a cluster Queue through spec.allowedNamespaces. Tenants can then manage their own hierarchy within that boundary. This separates resource-pool authorization from day-to-day tenant queue management. Different namespaces can use the same queue names without sharing queue identity. The feature is disabled by default. The Helm switch enables the participating controller, scheduler, and admission components and selects hierarchical capacity when using the chart’s default scheduler configuration:

helm upgrade --install volcano volcano-sh/volcano \
  --namespace volcano-system --create-namespace --version 1.16.0 \
  --set custom.namespace_queue_enable=true

Create the namespace and authorized parent Queue, then the tenant queue:

apiVersion: v1
kind: Namespace
metadata:
  name: team-a
---
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: research
spec:
  capability:
    cpu: "100"
    memory: "200Gi"
  deserved:
    cpu: "100"
    memory: "200Gi"
  allowedNamespaces:
  - team-a
---
apiVersion: scheduling.volcano.sh/v1beta1
kind: NamespaceQueue
metadata:
  name: training
  namespace: team-a
spec:
  parent: cluster/research
  capability:
    cpu: "40"
    memory: "80Gi"
  deserved:
    cpu: "40"
    memory: "80Gi"
  reclaimable: true

Grant tenant RBAC permissions for namespacequeues in the tenant’s namespace. After the NamespaceQueue reports Ready=True and Authorized=True, Jobs and PodGroups in team-a can reference it with:

spec:
  queue: namespace/training

Bare names still reference cluster Queues. cluster/ identifies a cluster Queue parent, while an unprefixed parent name identifies a NamespaceQueue in the same namespace. Initial scheduling support uses hierarchical capacity. Do not enable proportion alongside it. Related:

• PRs: #5590, #5629
• User guide: NamespaceQueue
• Design: Namespace-scoped Queues
• Contributor: @gitGurugu

3. fairshare improves long-term tenant fairness

Volcano traditionally organizes resource sharing and quota governance around cluster-scoped Queues, which fits deployments that assign a dedicated queue to each tenant. Other platforms use namespaces as the isolation boundary for teams or projects, with multiple tenants submitting into one Queue. These deployments need fairness within that shared queue as well as fairness across queues. Current-allocation snapshots alone do not capture resource consumption over time. If a namespace keeps submitting GPU jobs, each completed job may be replaced by another from the same namespace. A tenant arriving later can wait even though the earlier tenant has already consumed substantial compute time. Inspired by Slurm fair-share scheduling, the new fairshare plugin identifies tenants by namespace and tracks cumulative resource-seconds for each namespace within a queue. Consumption decays exponentially, with a configurable half-life that defaults to four hours. Namespaces with lower historical consumption receive earlier consideration, while old usage gradually loses influence. Fairshare complements DRF’s current-resource fairness. It influences pending-job ordering and can optionally gate enqueue admission. It does not preempt jobs that are already running. Operators can choose target queues and resource types and optionally persist usage state to a ConfigMap across scheduler restarts. Example scheduler configuration:

actions: "enqueue, allocate, backfill"
tiers:
- plugins:
  - name: priority
  - name: gang
  - name: conformance
  - name: fairshare
    arguments:
      fairshare.targetQueues: "gpu-queue"
      fairshare.resourceKey: "nvidia.com/gpu"
      fairshare.halfLifeMinutes: "240"
      fairshare.enableEnqueueGate: "false"
      fairshare.persistState: "false"
- plugins:
  - name: drf
  - name: capacity
  - name: predicates
  - name: nodeorder

Put fairshare before drf so historical usage can influence comparisons before DRF decides them. Keep priority first when PriorityClass should take precedence. State persistence and enqueue gating are disabled by default. Related:

• Feature PR: #5161
• Design: Fair Share Scheduling
• User guide: Fairshare Plugin
• Feature contributor: @dmildh-absci

4. PodGroup topology anti-affinity

Keeping each workload instance within one topology domain does not ensure that different instances use separate domains. For example, multiple inference replicas can each satisfy their own network locality constraints but still land in the same supernode. A failure in that domain could affect all instances. The group-topology-affinity plugin adds required and preferred PodGroup anti-affinity across HyperNode domains. Rules select peer PodGroups by their labels and can optionally select namespaces. They compose with networkTopology, allowing each group to stay together while different groups use separate domains. Add these entries to the existing scheduler plugin list, retaining its other actions and plugins:

- name: group-topology-affinity
  arguments:
    weight: 10
- name: network-topology-aware
  arguments:
    weight: 10

With a HyperNode tree containing the supernode tier, configure each participating PodGroup as follows, changing the name for each instance:

apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
  name: inference-a
  namespace: default
  labels:
    app: model-a
spec:
  minMember: 2
  queue: default
  networkTopology:
    mode: hard
    highestTierName: supernode
  topologyAffinity:
    podGroupAntiAffinity:
      required:
      - podGroupSelector:
          matchLabels:
            app: model-a
        topologyTierName: supernode

Member Pods must use schedulerName: volcano and the scheduling.k8s.io/group-name annotation. Rules match PodGroup labels, not Pod labels. A term specifies either topologyTierName or topologyTier. Preferred rules additionally use a weight from 1 to 100. Enforcement applies to allocate. BestEffort Pods placed through backfill do not receive this isolation guarantee. SubGroup affinity and anti-affinity, cross-PodGroup affinity, and automatic relocation of running Pods are outside the implemented scope. Related:

• PR: #5349
• Design: PodGroup Topology Affinity
• User guide: PodGroup Anti-Affinity
• Contributor: @wangyang0616

5. Expanding th