Kubernetes Mixed Version Proxy Moves to Beta: What SMBs Running Multi-Master Clusters Need to Know

Why this matters

Managing Kubernetes clusters with multiple API servers running different versions is common during upgrades to maintain high availability. For SMBs, especially those in healthcare and professional services with compliance demands, ensuring these upgrades do not disrupt workloads or cause data inconsistencies is a critical concern. Without a mechanism to coordinate requests across mixed-version API servers, clients might receive incorrect 404 Not Found errors when querying newer API resources that some servers do not yet recognize.

This problem is more than an inconvenience—it can lead to unintended side effects such as improper garbage collection or failure to delete namespaces, which may cause cascading operational issues. The Mixed Version Proxy (MVP) feature, introduced in Kubernetes 1.28 as an alpha and now graduating to beta in 1.36, addresses this gap by transparently proxying requests to the appropriate API server that can handle them.

By enabling MVP, clusters gain a more accurate, unified view of available APIs during rolling upgrades, reducing the risk of errors caused by version skew. This is particularly essential for SMBs that rely on predictable cluster behavior to meet uptime and compliance requirements without dedicating extensive operational resources to troubleshooting upgrade problems.

What usually goes wrong

When a Kubernetes cluster is undergoing an upgrade, each control plane node might run a different version of the API server for a short period. Each API server exposes varying sets of API groups, versions, and resources depending on its version. If a client request lands on an API server that does not support the requested resource version, the server returns a 404 Not Found error, even though the resource exists elsewhere in the cluster.

This incorrect response can confuse clients and controllers, causing them to think the resource is missing. For example, namespace deletion controllers may block the removal of namespaces because they cannot correctly assess resource presence across mixed-version API servers. Similarly, garbage collection processes might mistakenly delete resources or leave stale ones behind.

Before MVP, teams often had to work around this by coordinating all clients to talk only to certain API servers or by halting upgrades until all nodes were updated. These approaches increase operational overhead, risk downtime, and add complexity to an already delicate process. Further, discovery APIs—used by clients to learn what APIs are available—would reflect only the capabilities of the single API server they contacted, not the entire cluster, leading to incomplete or misleading views.

A better Cloudain-style approach

The Mixed Version Proxy offers a pragmatic solution by enabling API servers to proxy requests they cannot handle locally to peer servers that can. This proxying respects secure communication via TLS and uses a discovery cache to dynamically identify which peer API servers serve which resources. The feature's transition from alpha to beta in Kubernetes 1.36 includes several architectural improvements that make it more reliable and easier to adopt.

Most notably, MVP no longer depends on the StorageVersion API, which had limitations with custom resources and aggregated APIs. Instead, it leverages aggregated discovery mechanisms to gather peer API server capabilities dynamically. This change broadens MVP’s compatibility with the diverse API landscape often present in real-world clusters.

Another significant enhancement is the introduction of peer-aggregated discovery. Now, when a client queries the API server for available APIs, the server merges its local data with the discovery information from all peers. The client thus receives a complete and unified API catalog regardless of which API server it connects to. This unified discovery eliminates confusion and reduces the need for custom client logic to handle version skew.

For secure operation, MVP requires that API servers are configured with specific flags, such as --peer-ca-file for TLS verification between peers and --peer-advertise-ip and --peer-advertise-port to specify network addresses for inter-API server communication. These configurations ensure that proxying is both secure and reliable, which is vital for production environments with strict compliance requirements.

A simple next step

For SMBs managing multi-master Kubernetes clusters, the simplest next step is to review your API server configurations as you prepare to upgrade to Kubernetes 1.36 or later. Confirm that the --peer-ca-file flag is set correctly with the appropriate CA bundle to avoid TLS errors in peer communication. If your cluster has complex network layouts, explicitly set the peer advertise IP and port flags to ensure API servers can reach each other securely.

Testing MVP in a staging environment before rolling it out in production is advisable. This practice allows teams to observe how the proxy handles requests for new API versions during an upgrade window, identify any potential issues with discovery aggregation, and verify that client applications behave as expected.

Additionally, familiarize yourself with the ability to request a non-aggregated discovery view by specifying the profile=nopeer parameter. This option can aid in troubleshooting or auditing the exact APIs served by a particular API server.

Implementing MVP does not require changes to client software, which is beneficial for SMBs with limited engineering bandwidth. The feature works transparently, improving cluster behavior during upgrades without demanding application-level modifications or complex routing rules.

How Cloudain can help

Cloudain’s expertise in Kubernetes platform engineering can guide SMBs through the nuances of adopting the Mixed Version Proxy feature safely and effectively. From reviewing API server configurations to designing upgrade strategies that minimize downtime and compliance risks, Cloudain can tailor recommendations to your cluster’s architecture and business priorities.

In addition, Cloudain can assist with validating cluster behavior in staging environments, ensuring that peer-aggregated discovery functions as intended and that proxying does not introduce unintended side effects. By integrating MVP readiness into your standard operating procedures, Cloudain supports a smoother upgrade experience that aligns with the operational realities of healthcare and professional services workloads.

Leveraging Cloudain’s practical, experience-driven guidance helps teams avoid common pitfalls with mixed-version API servers and makes version upgrades a more manageable aspect of Kubernetes lifecycle management.