Use Pod-Level Resources with kubelet Resource Managers
Kubernetes Docs · نسخهٔ f284a9e7c87f · CC BY 4.0 برای مستندات و مجوزهای همراه
<!-- overview -->
{{< feature-state feature_gate_name="PodLevelResourceManagers" >}}
This tutorial demonstrates how to configure the kubelet's resource managers (topology, CPU, and memory) to support pod-level resource specifications. You can define *hybrid allocation models* where some containers receive exclusive, NUMA-aligned infrastructure resources while others share the remaining resources from a pod-level shared pool.
To learn more about the concepts behind this feature, read the Pod-level resource managers concept page.
<!-- lessoncontent -->
{{% heading "objectives" %}}
- Configure kubelet resource managers (CPU, memory, and topology) to support
pod-level resources.
- Deploy workloads using the
podTopology Manager scope to achieve
single-NUMA alignment with mixed exclusive and shared containers.
- Inspect and verify how CPU and memory resources are partitioned between
exclusive containers and the pod shared pool.
- Understand admission rejection rules when a pod-level shared pool would be
empty.
- Deploy workloads using the
containerTopology Manager scope with mixed
container allocations.
{{% heading "prerequisites" %}}
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
To complete this tutorial, you need:
- A Kubernetes cluster with Linux worker nodes (pod-level resource
managers are not supported on Windows nodes).
- At least one worker node with NUMA topology (preferably multiple NUMA
nodes to observe alignment).
- Administrative access (root or
sudo) on the worker node(s) to modify
[kubelet configuration](/docs/reference/config-api/kubelet-config.v1beta1/) and restart the kubelet service.
kubectlaccess with permission to create namespaces and pods.
Ensure the following feature gates are enabled for your control plane and for the worker nodes:
PodLevelResourcesPodLevelResourceManagers
Create a namespace {#create-namespace}
Create a namespace so that the resources created in this tutorial are isolated from the rest of your cluster:
kubectl create namespace plrm-tutorialUse pod scope with mixed allocation {#use-pod-scope-with-mixed-allocation}
When the Topology Manager scope is set to pod, the kubelet performs a single NUMA alignment for the entire Pod based on .spec.resources. The resulting resource budget is then partitioned: containers requesting Guaranteed resources receive exclusive slices, while containers that do not receive an exclusive allocation share the remaining budget in a pod-level shared pool.
Step 1: Configure kubelet for pod scope
To enable this behavior, configure the kubelet on the target worker node(s) where you want to run these workloads with the required policies. You can update your kubelet configuration for those nodes as follows:
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cpuManagerPolicy: "static"
memoryManagerPolicy: "Static"
topologyManagerScope: "pod"
topologyManagerPolicy: "single-numa-node"- For
topologyManagerPolicy, the valid values are
single-numa-node, restricted, or best-effort. You cannot specify any other value when using Pod-level resource management.
Restart the kubelet to apply the configuration. For example, on Linux with systemd: systemctl restart kubelet.service.
Step 2: Deploy a mixed-allocation pod
Consider the following example Pod manifest. The Pod requests a total budget of 4 CPUs at the pod level (.spec.resources). Inside the Pod:
- the
main-appcontainer requests an exclusive allocation of 2 entire CPU
cores (requests = limits = 2 CPU).
- because
metrics-sidecarandlogging-sidecardo not specify
container-level requests, those two sidecar containers share the CPU cores that remain from the pod-level shared pool: 2 CPU cores.
{{% code_sample file="pods/resource/pod-level-resource-managers-pod-scope-mixed.yaml" %}}
Apply the manifest to your cluster:
kubectl apply -f https://k8s.io/examples/pods/resource/pod-level-resource-managers-pod-scope-mixed.yaml --namespace=plrm-tutorialStep 3: Verify and inspect resource partitioning
- Check that the Pod is running successfully:
kubectl get pod pod-scope-mixed --namespace=plrm-tutorial- Understand what happened behind the scenes:
flowchart TD
subgraph Pod["Pod-Level Budget: 4 CPUs, 4Gi Memory"]
direction TB
C1["main-app<br/>(Exclusive: 2 CPUs, 2Gi Memory)"]
subgraph Pool["Pod Shared Pool: 2 CPUs, 2Gi Memory"]
C2["metrics-sidecar"]
C3["logging-sidecar"]
end
end- Pod Alignment: The Topology Manager evaluated the 4 CPU pod request
(spec.resources) and assigned the entire Pod to a single NUMA node.
- Exclusive Allocation: The CPU Manager allocated a dedicated 2 CPU slice
to main-app. CPU CFS quota throttling is disabled for main-app, giving it unthrottled access to those exclusive cores.
- Pod Shared Pool: The remaining 2 CPUs form a pod-level shared pool.
metrics-sidecar and logging-sidecar specify no container-level resources (resources: {}), so they run within this pod-isolated shared pool with CFS quota enforcement enabled. While shared between these two sidecars, the pool is isolated from external workloads on the node, providing the sidecars with dedicated NUMA locality and protection from node-level resource contention.
Observe empty shared pool admission restrictions {#empty-shared-pool-restrictions}
When using the pod scope, kubelet admission control rejects Pod specifications that would result in an empty Pod shared pool when there are containers that require one.
If the sum of exclusive resource requests from Guaranteed containers equals the total pod-level budget, and at least one other container requires the shared pool, the kubelet rejects the Pod.
Step 4: Examine the invalid pod manifest
Consider the following manifest. The Pod requests a total budget of 4 CPUs. container-a requests an exclusive 1 CPU and container-b requests an exclusive 3 CPUs (totaling 4 CPUs). container-c does not request exclusive resources and requires a shared pool, but 0 CPUs remain:
{{% code_sample file="pods/resource/pod-level-resource-managers-empty-shared-pool.yaml" %}}
Step 5: Attempt deployment and observe rejection
- Apply the manifest:
kubectl apply -f https://k8s.io/examples/pods/resource/pod-level-resource-managers-empty-shared-pool.yaml --namespace=plrm-tutorial- Inspect the Pod events to observe the admission error:
kubectl describe pod empty-shared-pool --namespace=plrm-tutorialNotice the event message explaining that the kubelet rejected the Pod because the pod-level shared pool would be empty for containers requiring shared resources:
Status: Failed
Reason: TopologyAffinityError
Message: Pod was rejected: Pod Scope pod with pod-level resources failed admission under pod-scope topology manager flowchart TD
subgraph Pod["Pod-Level Budget: 4 CPUs, 4Gi Memory"]
direction TB
C1["container-a<br/>(Exclusive: 1 CPU, 1Gi Memory)"]
C2["container-b<br/>(Exclusive: 3 CPUs, 3Gi Memory)"]
subgraph Pool["Pod Shared Pool: 0 CPUs, 0Gi Memory"]
C3["container-c<br/>(Requires shared pool)"]
end
end
Pool --> Rejection["Admission Error: Pod Rejected!"]
style Rejection fill:#ffcccc,stroke:#ff0000,stroke-width:2pxUse container scope with mixed allocation {#use-container-scope-with-mixed-allocation}
You can also configure the Topology Manager scope to container. In this mode, the kubelet evaluates each container individually for exclusive allocation, while the overall Pod budget in .spec.resources still enforces QoS and cgroup limit boundaries.
Step 6: Configure kubelet for container scope
Update your kubelet configuration on the target worker node(s) for the container scope:
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cpuManagerPolicy: "static"
memoryManagerPolicy: "Static"
topologyManagerScope: "container"
topologyManagerPolicy: "single-numa-node"Restart the kubelet to apply the configuration. For example, on Linux with systemd: systemctl restart kubelet.service.
Step 7: Deploy a container-scoped mixed workload
Consider the following example Pod manifest. The Pod has a total budget of 4 CPUs:
infrastructure-sidecarrequests an exclusive 2 CPU slice (requests=
limits = 2 CPU).
worker-1andworker-2do not specify container-level requests and run in
the general, node-wide shared pool.
{{% code_sample file="pods/resource/pod-level-resource-managers-container-scope-mixed.yaml" %}}
Apply the manifest:
kubectl apply -f https://k8s.io/examples/pods/resource/pod-level-resource-managers-container-scope-mixed.yaml --namespace=plrm-tutorialStep 8: Verify the deployment
- Check that the Pod is running:
kubectl get pod container-scope-mixed --namespace=plrm-tutorial- Understand what happened behind the scenes:
flowchart TD
subgraph Pod["Pod-Level Budget: 4 CPUs, 4Gi Memory"]
direction TB
C1["infrastructure-sidecar<br/>(Exclusive NUMA Slice: 2 CPUs, 2Gi Memory)"]
subgraph NodePool["Node Shared Pool: Pod-level limit"]
C3["worker-2"]
C2["worker-1"]
end
end- Container-Scoped Alignment: Under
containerscope, thekubelet
evaluates containers individually. infrastructure-sidecar receives an exclusive, NUMA-aligned 2 CPU slice directly from the node's allocatable pool.
- Node Shared Pool:
worker-1andworker-2specify no container-level
resource requests (resources: {}), so under container scope they run in the node's general shared pool (rather than a pod-isolated pool).
- Pod Limit Enforcement: The total CPU consumption across all containers
remains capped at 4 CPUs by the pod-level limit (spec.resources.limits).
{{% heading "cleanup" %}}
Delete the namespace and all sample Pods created during this tutorial:
kubectl delete namespace plrm-tutorial{{% heading "whatsnext" %}}
- Read the concept documentation:
- Pod-level resource managers
- Learn how to configure node-level resource managers:
- Control Topology Management Policies on a Node
- Control CPU Management Policies on the Node
- Utilizing the NUMA-aware Memory Manager
- Learn how to assign resources:
- Assign Pod-level CPU and memory resources
- Assign CPU Resources to Containers and Pods
- Assign Memory Resources to Containers and Pods