HAMi: Kubernetes-Native GPU Virtualization and Heterogeneous Accelerator Scheduling

How HAMi shares GPUs, enforces memory limits, and schedules across vendors without changing your application code

Your team has a pool of A100s, but each inference job only needs 3GB of memory. You are either wasting most of the GPU or fighting over who gets the whole card. The standard answer—one pod per GPU—leaves utilization in the single digits, and the usual workaround, time-slicing, gives you no memory isolation at all.

HAMi is a CNCF Incubating project that addresses exactly this problem. It is a Kubernetes-native device virtualization and scheduling middleware that lets you slice GPUs by memory and compute, enforce those limits inside the container, and schedule across NVIDIA, Ascend, and other accelerators through a single layer. It does not change your application code—the in-container library intercepts CUDA calls via LD_PRELOAD.

The codebase is roughly 100k lines of Go, with an active community and regular releases. By the end of this article you will understand how HAMi’s control plane (scheduler extender, mutating webhook) and data plane (device plugin, vGPUmonitor) work together, and where its design decisions pay off or cost you.

What HAMi Is (and Isn’t)

HAMi is Kubernetes middleware that virtualizes GPUs and other accelerators, letting pods request fractions of a device—memory and compute cores—rather than whole cards. It schedules those fractional requests with device-aware policies such as binpack, spread, and topology-aware placement, and enforces per-workload memory limits in-container. The project is CNCF Incubating, actively maintained, and shipped v2.10.0 recently.

HAMi is not a GPU driver, a container runtime, or a replacement for kube-scheduler. It layers on top of the NVIDIA driver and containerd, and it works alongside kube-scheduler as an extender rather than substituting for it. The mutating webhook, scheduler extender, device plugins, and in-container libraries are all additive components that integrate with the standard Kubernetes control plane.

The system sits between Kubernetes—the API server, scheduler, and kubelet—and the accelerator drivers. Workloads and higher-level batch schedulers like Volcano operate above it; device plugins, the container runtime, and hardware drivers sit below. This position lets HAMi present a unified resource model across heterogeneous accelerators—NVIDIA, Ascend, Cambricon, Hygon, and others—through a single scheduling and allocation workflow, without requiring application code changes.

Architecture: Control Plane and Data Plane

HAMi splits its architecture into two planes. The control plane makes allocation decisions; the data plane executes them on nodes. A mutating webhook, scheduler extender, device abstraction layer, and node lock utility form the control plane. The data plane consists of the device plugin, vGPUmonitor, and the in-container libvgpu library.

The pod’s journey begins at admission. The mutating webhook intercepts the pod, injects device resource requests, and validates constraints—denying pods with privileged containers that request devices. The webhook also checks resource quotas and can overwrite the default scheduler name.

Next, kube-scheduler calls the scheduler extender’s /filter endpoint via HTTP POST. The scheduler core computes device fit per node, scores nodes based on binpack/spread policy and device topology, then selects the best node. Before binding, the scheduler acquires a per-node lock via the node lock utility, which uses a node annotation with timestamp and pod identity to prevent concurrent allocation conflicts. The scheduler writes the allocation to pod annotations and patches the pod through the /bind endpoint.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

sequenceDiagram
    participant User
    participant APIServer
    participant Webhook
    participant KubeScheduler
    participant SchedulerExtender
    participant Kubelet
    participant DevicePlugin

    User->>APIServer: Submit pod with nvidia.com/gpu: 1
    APIServer->>Webhook: Admission review (HTTP)
    Webhook->>APIServer: Mutated pod + injected resources
    KubeScheduler->>SchedulerExtender: POST /filter (HTTP)
    SchedulerExtender->>SchedulerExtender: Filter nodes, score, lock node
    SchedulerExtender->>APIServer: Patch pod with allocation annotation
    KubeScheduler->>SchedulerExtender: POST /bind (HTTP)
    APIServer->>Kubelet: Pod scheduled to node
    Kubelet->>DevicePlugin: Allocate() (gRPC)
    DevicePlugin->>Kubelet: Env vars, mounts, device specs

The device abstraction layer in pkg/device/devices.go defines the Devices interface that all vendor backends implement. This interface centralizes annotation encoding/decoding, resource request generation, and common fit/score logic. Each backend—NVIDIA, Ascend, Cambricon, Hygon, and others—implements this interface, allowing one scheduler to handle heterogeneous accelerators through a unified resource model.

On the data plane, the NVIDIA device plugin is forked from NVIDIA/k8s-device-plugin. It registers devices with the kubelet and serves the DevicePlugin gRPC API. When the kubelet calls Allocate(), the plugin sets up the container runtime environment based on pod annotations. The resource manager inside the plugin validates requests against sharing strategies and manages device enumeration and health checks.

The vGPUmonitor collects per-container GPU metrics by reading shared-memory cache files written by libvgpu, the in-container library injected via LD_PRELOAD. libvgpu intercepts CUDA memory allocations to enforce per-workload limits.

Key Features: From Memory Limits to Dynamic MIG

HAMi’s core value proposition is device sharing with hard isolation. When a pod requests nvidia.com/gpu: 1 and nvidia.com/gpumem: 3000, the scheduler places it on a GPU with sufficient free memory, and the in-container libvgpu library intercepts CUDA allocation calls to enforce that 3000 MiB ceiling. This prevents a misbehaving job from exhausting GPU memory and OOM-killing neighboring containers on the same physical device. Compute sharing works similarly: pods request a fraction of GPU cores, and the library throttles kernel execution to match.

Scheduling policies are controlled per-pod via the hami.io/gpu-scheduler-policy annotation. The available policies—binpack, spread, topology-aware, mutex, and NUMA-aware—map to concrete placement strategies in pkg/scheduler/policy/gpu_policy.go and node_policy.go. Binpack packs workloads onto the fewest devices to maximize idle nodes; spread does the opposite for fault tolerance. Topology-aware scoring uses GPU pair links to place multi-GPU pods on devices with high interconnect bandwidth. The mutex policy only allocates idle GPUs, giving a pod exclusive access when requested. NUMA-aware placement considers CPU-GPU proximity for latency-sensitive workloads. A shared policy layer applies scoring weights, so backends implement policy-neutral scores and the scheduler inverts or weights them per policy.

Dynamic MIG is a notable departure from static MIG configuration. Rather than resharding a physical GPU into fixed profiles at node startup, the device plugin creates MIG instances on demand per task, tracked by a MigInstanceManager. This gives hardware-level isolation for workloads that need it without requiring operators to pre-partition every GPU in the cluster.

Resource quotas operate at the namespace level. The mutating webhook checks per-namespace limits on accelerator memory and cores before admitting a pod, preventing one team from consuming the entire cluster’s accelerator pool. Quota enforcement happens at admission time, so rejected pods fail fast with a clear reason.

Per-container metrics flow through vGPUmonitor, which reads shared-memory cache files written by libvgpu and exposes Prometheus metrics for memory usage, utilization, and MIG information. This enables chargeback and capacity planning for shared clusters—something plain time-slicing cannot provide.

Heterogeneous support is the architectural payoff of the Devices interface in pkg/device/devices.go. NVIDIA, Ascend, Cambricon, Hygon, and a dozen other vendors implement the same interface, so one scheduler handles all of them with identical filtering, scoring, and binding logic. Adding a new accelerator means implementing the interface, not forking the scheduler.

Interface and Usage: Fractions, Policies, and Annotations

HAMi exposes a Kubernetes-native interface. Users request accelerator resources through standard pod resource limits, and control scheduling behavior through pod annotations. The simplest example, from examples/nvidia/default_use.yaml, requests one physical GPU with a 3000 MiB memory slice:

1
2
3
4

resources:
  limits:
    nvidia.com/gpu: 1
    nvidia.com/gpumem: 3000

The nvidia.com/gpu limit requests a number of physical NVIDIA GPUs; nvidia.com/gpumem requests memory in MiB. HAMi’s scheduler places the pod on a GPU with at least 3000 MiB free, allowing multiple pods to share the same physical device.

Scheduling behavior is controlled by two key annotations. hami.io/gpu-scheduler-policy selects among binpack, spread, mutex, topology-aware, and numa policies. The hami.io/device-scoring-weights annotation tunes per-pod scoring weights for slots, cores, and memory, e.g. slot=1,core=1,memory=3.

Deployment is Helm-based. After labeling nodes with gpu=on, install with:

1
2
3

helm repo add hami-charts https://project-hami.github.io/HAMi/
helm repo update
helm install hami hami-charts/hami -n kube-system

The scheduler runs an HTTP server registered in cmd/scheduler/main.go with routes for /filter, /bind, /webhook, /healthz, and /readyz. The /filter and /bind endpoints implement the kube-scheduler extender protocol; /metrics on port 9395 exposes scheduler metrics, while vGPUmonitor serves container-level GPU metrics on port 9394.

The mutating webhook intercepts pod admission and enforces constraints. It denies pods with privileged containers that request devices, and can overwrite the default scheduler name.

Use Cases: Where HAMi Shines and Where It Doesn’t

HAMi is a strong fit for teams operating shared GPU pools where many small inference workloads compete for a handful of expensive accelerators. A pod can request a fractional memory allocation—say 3 GiB of an A100—and HAMi will schedule multiple pods onto the same physical device. This directly addresses the utilization problem that arises when whole-GPU allocation leaves most of the memory idle.

Organizations running a heterogeneous fleet of NVIDIA, Ascend, and Cambricon accelerators benefit from HAMi’s single scheduling layer. The device abstraction interface in pkg/device/devices.go lets one scheduler handle all vendors through a common resource model, so platform teams avoid maintaining separate scheduling paths per hardware type.

The in-container libvgpu library enforces hard memory limits by intercepting CUDA allocation calls. This prevents a misbehaving job from exhausting GPU memory and OOM-killing neighboring containers on the same device—a real concern in shared environments where one workload’s leak affects everyone else.

For workloads that need a whole GPU, HAMi supports a mutex scheduling policy that only allocates idle devices, and requesting 100% cores implies exclusivity. Both mechanisms give users a path to whole-GPU semantics without disabling the sharing layer.

The fit is partial when using HAMi with Volcano for gang scheduling. HAMi integrates with Volcano and other schedulers, but it is not a batch scheduler itself—it handles device allocation, not job queuing or gang semantics. Teams needing those features must run Volcano alongside it.

HAMi is a poor fit where hardware-level isolation is mandatory. NVIDIA MIG provides stronger fault isolation and fixed partitioning profiles; HAMi’s software-based sharing cannot match that guarantee. It is also not a replacement for a full batch scheduler. Treat HAMi as a complementary device-virtualization layer, not a substitute for either MIG or Volcano.

How HAMi Compares to Alternatives

The table below positions HAMi against the three alternatives most commonly considered for GPU sharing on Kubernetes: NVIDIA MIG, NVIDIA time-slicing via the k8s-device-plugin, and Volcano. The comparison reflects general knowledge of these projects and may be out of date; where the analysis did not provide a value, the cell reads “unknown.”

The key differentiator is the isolation mechanism. MIG partitions the GPU in hardware, giving strong isolation at the cost of fixed profiles and NVIDIA-only support. Time-slicing shares the GPU without any memory isolation, so one container can exhaust device memory and affect neighbors. HAMi sits between these: it is a software layer that enforces memory limits via the in-container libvgpu library, works across vendors, but provides weaker isolation than hardware partitioning.

HAMi and Volcano are not competitors in the same dimension. Volcano is a batch scheduler that handles gang scheduling and queue management; HAMi is a device virtualization layer. They are complementary and can be used together, with Volcano handling job-level scheduling and HAMi managing device allocation within the cluster.

Under the Hood: The Device Abstraction and Node Lock

The scheduler’s ability to handle NVIDIA, Ascend, Cambricon, and other accelerators through one code path rests on the Devices interface in pkg/device/devices.go. Each backend implements this interface, exposing DeviceUsage (current memory and core consumption per device), DeviceInfo (per-device capacity and health), and PodDevices (the allocation state for a given pod). The interface’s Fit method determines whether a pod’s resource request can be satisfied by a node’s devices, while Score ranks candidate devices for placement. Annotation encoding and decoding are also centralized here, so the scheduler writes allocation state to pod annotations through the same interface regardless of vendor.

Because the scheduler core only depends on this interface, adding a new accelerator vendor means implementing the interface for that vendor’s backend—no changes to the scheduling loop itself. The webhook, filter, and bind paths all operate on the abstracted device model.

Concurrent allocation requests for the same node present a race condition: two pods could be scored against the same free device and both bind. HAMi addresses this with a per-node lock stored as a node annotation. The nodelock package in pkg/util/nodelock/nodelock.go implements the protocol:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

const (
	NodeLockKey = "hami.io/mutex.lock"
	NodeLockSep = ","
)

func LockNode(nodeName string, lockname string, pods *corev1.Pod) error {
	ctx := context.Background()
	node, err := client.GetClient().CoreV1().Nodes().Get(ctx, nodeName, metav1.GetOptions{})
	if err != nil {
		return err
	}
	if _, ok := node.Annotations[NodeLockKey]; !ok {
		return SetNodeLock(nodeName, lockname, pods)
	}
	lockTime, ns, previousPodName, err := ParseNodeLock(node.Annotations[NodeLockKey])
	if err != nil {
		return err
	}
	// ... expiration and owner checks ...
	return fmt.Errorf("node %s has been locked within %v: %w", nodeName, NodeLockTimeout, ErrNodeLockContention)
}

The lock value encodes a timestamp, namespace, and pod name separated by commas. LockNode first checks whether the annotation exists; if not, it acquires the lock via SetNodeLock, which patches the node with the lock annotation using a merge patch that includes the node’s resourceVersion for optimistic concurrency. If the lock exists, the function parses it, checks whether it has expired (default timeout is five minutes, configurable via HAMI_NODELOCK_EXPIRE), and verifies the owner pod still exists. A pod requesting devices from multiple vendors calls LockNode once per vendor, so the code treats a lock already held by the same pod as acquired rather than contending with itself.

The design is deliberately simple and observable: the lock state is visible in the node object, and any operator can see which pod holds it. The tradeoff is that it relies on API server consistency for correctness and serializes all allocation attempts for a node through a single annotation patch, which can become a bottleneck under heavy contention. The in-memory nodeLockManager mitigates cross-node contention by keeping separate mutexes per node, but the annotation patch itself remains the serialization point.

What to take away

HAMi demonstrates a workable pattern for device virtualization in Kubernetes: keep the control plane and data plane separate, communicate allocation state through pod annotations, and enforce limits in the container via library interposition. The Devices interface in pkg/device/devices.go is the linchpin—it lets one scheduler core handle many accelerator vendors without per-vendor scheduling logic. If you are building similar infrastructure, that interface boundary is the design worth copying.

The project also shows the cost of that generality. Backends vary in feature parity; some have hardcoded device memory values, and memory units differ across vendors. The nodelock utility hardcodes a single annotation key, and the scheduler extender is limited to the filter/bind API—no pre-score or reserve hooks. The in-container libvgpu behavior and MIG allocation internals are not fully documented in the codebase, so production adoption will require reading the source.

What remains genuinely useful: annotation-based allocation state makes the system debuggable with kubectl get pod -o yaml, and the RBAC static analysis tool in hack/tools/rbaccheck is a practical answer to manifest drift. The repository is at https://github.com/Project-HAMi/HAMi; the code is Apache-2.0 and actively maintained.

What this analysis could not determine

• The exact behavior and performance of the in-container libvgpu library (not fully in the digest).
• The full list of supported device backends and their feature parity (some backends are truncated).
• The specifics of the MIG instance manager’s allocation algorithm (migmgr.go omitted).
• How the scheduler handles node failures and device health transitions in detail (health.go omitted).
• The exact Helm chart configuration options and default values (values.yaml not fully shown).

Further diagrams