Extend Service IP Ranges
Kubernetes Docs · نسخهٔ f284a9e7c87f · CC BY 4.0 برای مستندات و مجوزهای همراه
<!-- overview --> {{< feature-state feature_gate_name="MultiCIDRServiceAllocator" >}}
This document shares how to extend the existing Service IP range assigned to a cluster.
{{% heading "prerequisites" %}}
{{< include "task-tutorial-prereqs.md" >}}
{{< version-check >}}
{{< note >}} While you can use this feature with an earlier version, the feature is only GA and officially supported since v1.33. {{< /note >}}
<!-- steps -->
Extend Service IP Ranges
The control plane ensures that there is a special ServiceCIDR object, with the well-known name kubernetes, that specifies an IP address range based on the value of the --service-cluster-ip-range command line argument to kube-apiserver.
kubectl get servicecidrNAME CIDRS AGE
kubernetes 10.96.0.0/28 17dThe well-known kubernetes Service, that exposes the kube-apiserver endpoint to the Pods, calculates the first IP address from the default ServiceCIDR range and uses that IP address as its cluster IP address.
kubectl get service kubernetesNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 17dThe default Service, in this case, uses the ClusterIP 10.96.0.1, that has the corresponding IPAddress object.
kubectl get ipaddress 10.96.0.1NAME PARENTREF
10.96.0.1 services/default/kubernetesThe ServiceCIDRs are protected with {{<glossary_tooltip text="finalizers" term_id="finalizer">}}, to avoid leaving Service ClusterIPs orphans; the finalizer is only removed if there is another subnet that contains the existing IPAddresses or there are no IPAddresses belonging to the subnet.
Extend the number of available IPs for Services
There are cases that users will need to increase the number addresses available to Services, previously, increasing the Service range was a disruptive operation that could also cause data loss. With this new feature users only need to add a new ServiceCIDR to increase the number of available addresses.
Adding a new ServiceCIDR
On a cluster with a 10.96.0.0/28 range for Services, there is only 2^(32-28) - 2 = 14 IP addresses available. The kubernetes.default Service is always created; for this example, that leaves you with only 13 possible Services.
for i in $(seq 1 13); do kubectl create service clusterip "test-$i" --tcp 80 -o json | jq -r .spec.clusterIP; done10.96.0.11
10.96.0.5
10.96.0.12
10.96.0.13
10.96.0.14
10.96.0.2
10.96.0.3
10.96.0.4
10.96.0.6
10.96.0.7
10.96.0.8
10.96.0.9
error: failed to create ClusterIP service: Internal error occurred: failed to allocate a serviceIP: range is fullYou can increase the number of IP addresses available for Services, by creating a new ServiceCIDR that extends or adds new IP address ranges.
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: ServiceCIDR
metadata:
name: newcidr1
spec:
cidrs:
- 10.96.0.0/24
EOFservicecidr.networking.k8s.io/newcidr1 createdand this will allow you to create new Services with ClusterIPs that will be picked from this new range.
for i in $(seq 13 16); do kubectl create service clusterip "test-$i" --tcp 80 -o json | jq -r .spec.clusterIP; done10.96.0.48
10.96.0.200
10.96.0.121
10.96.0.144Deleting a ServiceCIDR
You cannot delete a ServiceCIDR if there are IPAddresses that depend on the ServiceCIDR.
kubectl delete servicecidr newcidr1servicecidr.networking.k8s.io "newcidr1" deletedKubernetes uses a finalizer on the ServiceCIDR to track this dependent relationship.
kubectl get servicecidr newcidr1 -o yamlapiVersion: networking.k8s.io/v1
kind: ServiceCIDR
metadata:
creationTimestamp: "2023-10-12T15:11:07Z"
deletionGracePeriodSeconds: 0
deletionTimestamp: "2023-10-12T15:12:45Z"
finalizers:
- networking.k8s.io/service-cidr-finalizer
name: newcidr1
resourceVersion: "1133"
uid: 5ffd8afe-c78f-4e60-ae76-cec448a8af40
spec:
cidrs:
- 10.96.0.0/24
status:
conditions:
- lastTransitionTime: "2023-10-12T15:12:45Z"
message: There are still IPAddresses referencing the ServiceCIDR, please remove
them or create a new ServiceCIDR
reason: OrphanIPAddress
status: "False"
type: ReadyBy removing the Services containing the IP addresses that are blocking the deletion of the ServiceCIDR
for i in $(seq 13 16); do kubectl delete service "test-$i" ; doneservice "test-13" deleted
service "test-14" deleted
service "test-15" deleted
service "test-16" deletedthe control plane notices the removal. The control plane then removes its finalizer, so that the ServiceCIDR that was pending deletion will actually be removed.
kubectl get servicecidr newcidr1Error from server (NotFound): servicecidrs.networking.k8s.io "newcidr1" not foundKubernetes Service CIDR Policies
Cluster administrators can implement policies to control the creation and modification of ServiceCIDR resources within the cluster. This allows for centralized management of the IP address ranges used for Services and helps prevent unintended or conflicting configurations. Kubernetes provides mechanisms like Validating Admission Policies to enforce these rules.
Preventing Unauthorized ServiceCIDR Creation/Update using Validating Admission Policy
There can be situations that the cluster administrators want to restrict the ranges that can be allowed or to completely deny any changes to the cluster Service IP ranges.
{{< note >}} The default "kubernetes" ServiceCIDR is created by the kube-apiserver to provide consistency in the cluster and is required for the cluster to work, so it always must be allowed. You can ensure your ValidatingAdmissionPolicy doesn't restrict the default ServiceCIDR by adding the clause:
matchConditions:
- name: 'exclude-default-servicecidr'
expression: "object.metadata.name != 'kubernetes'"as in the examples below. {{</ note >}}
Restrict Service CIDR ranges to some specific ranges
The following is an example of a ValidatingAdmissionPolicy that only allows ServiceCIDRs to be created if they are subranges of the given allowed ranges. (So the example policy would allow a ServiceCIDR with cidrs: ['10.96.1.0/24'] or cidrs: ['2001:db8:0:0:ffff::/80', '10.96.0.0/20'] but would not allow a ServiceCIDR with cidrs: ['172.20.0.0/16'].) You can copy this policy and change the value of allowed to something appropriate for you cluster.
{{% code_sample file="service/networking/vap-servicecidr-default.yaml" %}}
Consult the CEL documentation to learn more about CEL if you want to write your own validation expression.
Restrict any usage of the ServiceCIDR API
The following example demonstrates how to use a ValidatingAdmissionPolicy and its binding to restrict the creation of any new Service CIDR ranges, excluding the default "kubernetes" ServiceCIDR:
{{% code_sample file="service/networking/vap-servicecidr-deny.yaml" %}}