Pod Quality of Service Classes
Kubernetes Docs · نسخهٔ f284a9e7c87f · CC BY 4.0 برای مستندات و مجوزهای همراه
<!-- overview -->
This page introduces _Quality of Service (QoS) classes_ in Kubernetes, and explains how Kubernetes assigns a QoS class to each Pod as a consequence of the resource constraints that you specify for the containers in that Pod. Kubernetes relies on this classification to make decisions about which Pods to evict when there are not enough available resources on a Node.
<!-- body -->
Quality of Service classes
Kubernetes classifies the Pods that you run and allocates each Pod into a specific _quality of service (QoS) class_. Kubernetes uses that classification to influence how different pods are handled. Kubernetes does this classification based on the resource requests of the {{< glossary_tooltip text="Containers" term_id="container" >}} in that Pod, along with how those requests relate to resource limits. This is known as {{< glossary_tooltip text="Quality of Service" term_id="qos-class" >}} (QoS) class. Kubernetes assigns every Pod a QoS class based on the resource requests and limits of its component Containers. QoS classes are used by Kubernetes to decide which Pods to evict from a Node experiencing Node Pressure. The possible QoS classes are Guaranteed, Burstable, and BestEffort. When a Node runs out of resources, Kubernetes will first evict BestEffort Pods running on that Node, followed by Burstable and finally Guaranteed Pods. When this eviction is due to resource pressure, only Pods exceeding resource requests are candidates for eviction.
Guaranteed
Pods that are Guaranteed have the strictest resource limits and are least likely to face eviction. They are guaranteed not to be killed until they exceed their limits or there are no lower-priority Pods that can be preempted from the Node. They may not acquire resources beyond their specified limits. These Pods can also make use of exclusive CPUs using the [static](/docs/tasks/administer-cluster/cpu-management-policies/#static-policy-configuration) CPU management policy.
Criteria
For a Pod to be given a QoS class of Guaranteed:
- Every Container in the Pod must have a memory limit and a memory request, both greater than zero.
- For every Container in the Pod, the memory limit must equal the memory request.
- Every Container in the Pod must have a CPU limit and a CPU request, both greater than zero.
- For every Container in the Pod, the CPU limit must equal the CPU request.
If instead the Pod uses Pod-level resources:
{{< feature-state feature_gate_name="PodLevelResources" >}}
- The Pod must have a Pod-level memory limit and memory request, and their values must be equal.
- The Pod must have a Pod-level CPU limit and CPU request, and their values must be equal.
Burstable
Pods that are Burstable have some lower-bound resource guarantees based on the request, but do not require a specific limit. If a limit is not specified, it defaults to a limit equivalent to the capacity of the Node, which allows the Pods to flexibly increase their resources if resources are available. In the event of Pod eviction due to Node resource pressure, these Pods are evicted only after all BestEffort Pods are evicted. Because a Burstable Pod can include a Container that has no resource limits or requests, a Pod that is Burstable can try to use any amount of node resources.
Criteria
A Pod is given a QoS class of Burstable if:
- The Pod does not meet the criteria for QoS class
Guaranteed. - At least one Container in the Pod has a memory or CPU request or limit,
or the Pod has a Pod-level memory or CPU request or limit.
BestEffort
Pods in the BestEffort QoS class can use node resources that aren't specifically assigned to Pods in other QoS classes. For example, if you have a node with 16 CPU cores available to the kubelet, and you assign 4 CPU cores to a Guaranteed Pod, then a Pod in the BestEffort QoS class can try to use any amount of the remaining 12 CPU cores.
The kubelet prefers to evict BestEffort Pods if the node comes under resource pressure.
Criteria
A Pod has a QoS class of BestEffort if it doesn't meet the criteria for either Guaranteed or Burstable. In other words, a Pod is BestEffort only if none of the Containers in the Pod have a memory limit or a memory request, and none of the Containers in the Pod have a CPU limit or a CPU request, and the Pod does not have any Pod-level memory or CPU limits or requests. Containers in a Pod can request other resources (not CPU or memory) and still be classified as BestEffort.
Memory QoS with cgroup v2
{{< feature-state feature_gate_name="MemoryQoS" >}}
Memory QoS uses the memory controller of cgroup v2 to manage memory throttling and protection in Kubernetes. It uses the Pod's QoS class to decide which cgroup settings to apply, but it does not change how Pods are classified.
The default kubelet configuration does not enable memory throttling or memory protection. To enable either behavior, configure memoryThrottlingFactor or memoryReservationPolicy, respectively.
Memory throttling
Memory throttling is controlled by the kubelet configuration field memoryThrottlingFactor. Its default value is nil, which means that the kubelet does not set memory.high. To enable memory throttling, set memoryThrottlingFactor to a value greater than 0 and less than or equal to 1.
For Burstable Pods, the kubelet uses memory.high to throttle memory allocation before the workload hits its hard limit (memory.max). The throttling threshold for a container is calculated as:
memory.high = requests + memoryThrottlingFactor * (limits - requests)For example, with memoryThrottlingFactor set to 0.9, a container with a 256 MiB request and a 1 GiB limit gets memory.high set to roughly 947 MiB. If a Burstable container has no memory limit, the kubelet uses node allocatable memory in place of the limit.
For BestEffort containers, which have no memory request or limit, the kubelet calculates memory.high using a request of zero and node allocatable memory in place of the limit. Guaranteed containers do not get memory.high because their requests equal their limits.
Configuring memory reservation
Memory reservation is controlled via the kubelet configuration field memoryReservationPolicy:
None(default): the kubelet does not setmemory.minormemory.lowfor
containers and pods. No memory is hard-locked by the kernel.
TieredReservation: the kubelet sets tiered memory protection based on the
Pod's QoS class:
- Guaranteed pods:
memory.minis set to memory requests. The kernel
will not reclaim this memory under any circumstances.
- Burstable pods:
memory.lowis set to memory requests. The kernel
preferentially retains this memory but may reclaim it under extreme pressure.
- BestEffort pods: no memory protection is set.
For example, the following kubelet configuration enables memory throttling with a factor of 0.9 and tiered memory protection:
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
memoryThrottlingFactor: 0.9
memoryReservationPolicy: TieredReservationYou can configure either field independently. Omit memoryThrottlingFactor so that the kubelet does not set memory.high, or set memoryReservationPolicy: None to disable memory protection.
Because cgroup v2 memory protection is hierarchical, the kubelet also configures the ancestor cgroups. It sets memory.min on the kubepods root cgroup to the sum of the memory requests for Guaranteed and Burstable Pods. It sets memory.low on the kubepods root cgroup and the Burstable QoS cgroup to the sum of the memory requests for Burstable Pods. Without this ancestor coverage, the per-Pod and per-container protection would be ineffective.
{{< caution >}} For a Guaranteed Pod, memory requests equal memory limits. With memoryReservationPolicy: TieredReservation, memory.min therefore equals memory.max. For a workload that uses a large page cache, the kernel might be unable to reclaim enough page cache before the cgroup reaches memory.max, which can result in an OOM kill. Size the memory limit to include sufficient headroom for page cache. {{< /caution >}}
Disabling or rolling back Memory QoS
To disable Memory QoS, set the MemoryQoS feature gate to false, ensure that memoryReservationPolicy is unset or set to None, and restart the kubelet. At startup, the kubelet resets memory.min and memory.low on the kubepods root cgroup, and memory.low on the Burstable QoS cgroup, to zero. Pod-level and container-level memory.min and memory.low values can remain, but they are ineffective because the corresponding ancestor protection is zero.
When Memory QoS is disabled, the kubelet sets memory.high to max whenever the container runtime applies a resource configuration to a new, restarted, or in-place resized container. An already running container that is not restarted or resized can retain its previous memory.high value until the next restart or in-place resize.
System requirements
Memory QoS requires Linux with cgroup v2. Kernel 5.9 or higher is recommended because memory.high throttling on older kernels can trigger a known livelock bug. If the MemoryQoS feature gate is enabled on an older kernel, the kubelet logs a warning at startup.
Some behavior is independent of QoS class {#class-independent-behavior}
Certain behavior is independent of the QoS class assigned by Kubernetes. For example:
- Any Container exceeding a resource limit will be killed and restarted by the kubelet without
affecting other Containers in that Pod.
- If a Container exceeds its resource request and the node it runs on faces
resource pressure, the Pod it is in becomes a candidate for eviction. If this occurs, all Containers in the Pod will be terminated. Kubernetes may create a replacement Pod, usually on a different node.
- The resource request of a Pod is equal to the sum of the resource requests of
its component Containers, and the resource limit of a Pod is equal to the sum of the resource limits of its component Containers.
- The kube-scheduler does not consider QoS class when selecting which Pods to
preempt. Preemption can occur when a cluster does not have enough resources to run all the Pods you defined.
- The QoS class is determined when the Pod is created and remains unchanged for the
lifetime of the Pod. If you later attempt an in-place resize that would result in a different QoS class, the resize is rejected by admission.
{{% heading "whatsnext" %}}
- Learn about resource management for Pods and Containers.
- Learn about Node-pressure eviction.
- Learn about Pod priority and preemption.
- Learn about Pod disruptions.
- Learn how to assign memory resources to containers and pods.
- Learn how to assign CPU resources to containers and pods.
- Learn how to configure Quality of Service for Pods.