Platform teams strive to create “golden paths” for engineers and promote common standards across the organization with effective internal developer platforms (IDPs). However, this effort often conflicts with accelerating innovation. Platform builders must consider how to implement consistency without sacrificing the autonomy developers require to push boundaries and deliver genuinely innovative solutions.
In the context of Kubernetes-native development, the tension between common standards and developer freedom often comes down to the architectural decisions that platform teams make when determining how their teams’ workloads run on Kubernetes. Platform engineers should consider a multitenant Kubernetes architecture with virtual clusters to preserve autonomy while enforcing standards. To do so, they must look beyond typical solutions to platform “building blocks” that eliminate the need to choose between customization and standardization.
Why Platform Engineering Goals Fall Short With Kubernetes
Organizations currently take two approaches to reconcile consistency and compliance with autonomy and innovation. The first is a single-tenant architecture, in which administrators create many clusters — one per team, per customer, and so on. Often, companies have hundreds or even thousands of clusters. Each cluster adds significant cost and requires continuous upkeep; the Kubernetes cluster itself is not functional without several tools and services running atop it. In many cases, this platform stack is larger and more costly than the workload being run by it! Worse, these clusters typically run even when nobody uses them, such as on weekends.
Teams choose this single-tenant route to preserve developer autonomy and boost security by separating each workload into its dedicated cluster. Ironically, this is very expensive and does not provide the desired autonomy. The central IT team typically retains complete control over the platform stack in each cluster. Thus, despite handing over an entire cluster, the developer or team receiving it likely will not be the administrator. RBAC will still restrict them and only have access to specific namespaces.
The other common method is to adopt Kubernetes multitenancy, which can reduce costs and streamline operations with shared clusters. Multitenancy is attractive because it eliminates the duplication of costly platform stacks, making preserving autonomy very challenging. Typically, administrators in a multitenant scenario are giving developers access to individual namespaces within the Kubernetes cluster. Since they no longer have a cluster and are restricted to a namespace, teams face serious obstacles to self-service and productivity. For example, if one tenant is an internal pre-production team that wants to install ArgoCD to test it for their new delivery workflow, they can’t because ArgoCD requires them to install a CRD. They must go back to the cluster admin and ask them to install ArgoCD, but what if there is already a different version of ArgoCD running?
Platform teams need a compromise between single-tenant individual clusters and namespace isolation. In 2021, we examined this problem and devised a solution: virtualizing Kubernetes to give platform builders a powerful “building block” for custom IDPs.
Platform Engineering Success With Virtual Clusters
As we sought a solution to multitenancy, our first breakthrough was realizing that Kubernetes is almost like a Linux host. Kubernetes has users, permissions, RBAC, and namespaces, just like a Linux host has users, folders, and permissions — but sharing a Linux host is difficult without virtualization. What if we added virtualization to Kubernetes? Beyond the nodes and containers, what if we virtualized the control plane itself? This is when we invented virtual clusters, which essentially put a control plane inside a container, inside a namespace of another cluster, and IT teams can hand out this virtual cluster to a tenant. This means the tenant’s KubeContext points to the API server running in the container, simulating a real cluster.
The tenant has complete autonomy and can serve as cluster admin without negatively impacting organizational consistency and standards. This is because the platform stack components — Cert Manager, Istio, OpenPolicy Agent, and so on — run in an underlying “host” cluster, while virtual clusters are launched inside tenant namespaces. Platform teams can elevate a tenant’s privileges and unlock self-service without actually elevating their privileges in the host cluster.
Virtual cluster multitenancy thus plays a key role in platform engineering success by making it possible to run multitenant isolated clusters. This combines standardization and security with maximum autonomy; teams have isolated environments tailored to their needs yet still adhere to organizational policies. Overall, the organization needs fewer traditional clusters, which reduces complexity and eases the management burden. Platform teams only have to configure and manage tools like Cert Manager or an ingress controller on a small number of host clusters and can then share these tools across their fleet of virtual clusters.
Meanwhile, engineers also gain full autonomy and room to experiment safely. They have full access inside the virtual cluster but no outside of it except for the shared tooling the platform team has made available. This is the elusive middle ground: lightweight and cost-effective, like namespaces, but with the developer’s freedom and strict isolation of separate clusters. Security becomes less of a concern because even if one tenant launches a faulty controller or introduces another vulnerability, it will only affect the virtual cluster. In contrast, traditionally, such an error could bring the real Kubernetes cluster down.
Empowering Developers While Promoting Consistency
Virtual clusters are one example of a necessary shift in platform engineering initiatives. To succeed at boosting developer productivity and enforcing consistency as systems become ever more complex, platform builders must get creative and look beyond the typical solutions. Rather than searching for a comprehensive IDP, they must seek out flexible “building blocks” that solve the real, pressing issues facing their teams today. This approach ultimately leads to increased productivity and more innovative, secure applications.
The post Rethinking Kubernetes Multitenancy: A Smarter Approach for Platform Engineers appeared first on The New Stack.
By virtualizing Kubernetes, platform engineers can streamline operations and reduce infrastructure sprawl.