When designing the libraries and platforms on which our applications are built, the uses can end up being very different from those we originally envisioned. Because of this, it’s generally wise to build software on the principles of forward compatibility and extensibility.
However, in software architecture, working out the best way for an application to be user-friendly and cater to unexpected use cases is a perennial problem. As a result, there is a constant push-pull between highly opinionated systems on the one hand, and more flexible, customizable options on the other.
We’ve seen this pattern play out many times. One example would be Cloud Foundry versus Kubernetes. Cloud Foundry was strongly opinionated and pleasant to use. But Kubernetes, though much more complex and less developer-friendly, has dominated orchestration for much of the last decade because it is so much more flexible.
“We replay these patterns constantly,” Matthew Yacobucci, an architect at NGINX, now part of F5, told The New Stack. “If you take our domain, and the products we create at F5 and NGINX, they are Swiss Army knives that do just about everything. The BIG-IP, especially, can manage Layer 2 all the way up to Layer 7.
“It can be hard to configure, but we feel that the emergent properties you get from extensibility are ultimately better for customers than very opinionated systems.”
A counterexample is our control planes and with good reason. We’re generally loath to allow our developers to directly inject a shared object into a router, for instance, because the security implications are alarming.
But what would happen if there was a way to do it safely? Yacobucci is keen to see extensibility extended to the control plane. And what’s more, he thinks that WebAssembly (Wasm) might offer a way of making it happen.
Kubernetes Controllers and the Gateway API
Kubernetes gives you a set of built-in controllers — service controller, pod controller, deployment controller, replica set controller — plus a set of core APIs. You can, of course, extend the core APIs, but you have to build some sort of custom controller to do so.
The Kubernetes Gateway API also allows for extending its core resources, but, as with custom controllers with CRDs, Gateway implementations will need a logic piece as well — a typed client to watch for the new resource for instance. But, in addition, further transformations to each respective data plane configuration will be necessary for some configurations to be valid for Envoy, NGINX, and HAProxy.
To understand how we got here, it is worth looking at what happened with the Kubernetes Gateway API forerunner, the Ingress API. It was designed to do one thing, and it did it very well, but we immediately saw companies building custom resources (CRDs) to try and extend its capabilities in a variety of ways.
Among other examples, Heptio created Contour, which added HTTPProxy to overcome Ingress limitations, and NGINX created VirtualServer and VirtualServerRoute. “They were all, independently, trying to work around the restrictions of Ingress,” Yacobucci told us.
The developers of the Kubernetes Gateway API, which recently hit general availability status, have come up with an ingenious, if somewhat experimental, solution. The Gateway API defines a Kubernetes object that augments the behavior of another object in a standard way as a Metaresource. (ReferenceGrant is just one example of this general type of Metaresource.)
The Gateway API also defines a pattern called Policy Attachment, which augments the behavior of an object by adding settings that can’t be described within the original spec for that object.
What about implementers of the API? “We have Metaresources and Policy Attachments, and there is a robust configuration language that you can transport into your data plane,” Yacobucci said. “But in many cases, we believe that logic will need to be injected into the system to honor their intent.
“With Gateway API, we start thinking not just about ingress controllers, but also larger platforms. This is a philosophy that we want to maintain, by supporting the heritage that NGINX and F5 have in terms of extensibility, and extending that into the Gateway API control plane, as we are working on our Gateway API-conformant NGINX Gateway Fabric project.”
Parallel Innovations
What NGINX is architecting isn’t the only example. Envoy Gateway Extensions, for example, allow the Envoy Gateway to be extended by means of an extension server. This makes use of one or more gRPC pre/post hooks inside Envoy Gateway to modify the underlying Proxy’s xDS resources. This is a similar idea, although it focuses on Route Filters first, and not Policy Attachments.
Taking this approach stays within the existing systems in the Envoy and Gateway API ecosystems. But it does have a time cost. In addition, Envoy only supports one Extension Server per deployment.
There are also some parallels to tools like Crossplane, and perhaps Kratix, although both are broader in scope. The Gateway API focuses on Kubernetes app and API connectivity — primarily communications to and from the cluster, with some interesting, but fairly nascent, efforts under the GAMMA initiative to cover communications within the cluster.
Some gaps remain, Yacobucci acknowledged. His goal would be for the Gateway API to have a one-time installation with subsequent self-service.
“I think of self-service here from the customer and user perspective,” he said. “The admin adds an ingress controller from which application developers can customize, with the addition of Metaresources and Metaresource coupled logic, to handle merges, transformations or whatever bespoke configuration case.”
Making Programmable Control Planes a Reality
As we noted previously, there are security risks with allowing programming logic at the control plane layer, and Yacobucci himself acknowledged the risks. “I built something like this when I worked at an XMPP company,” he told us. “But it was extremely dangerous because you were in the same memory space [as the router], and the stanzas were mutating as they flowed through the system.”
However, he suggested, the strengths of WebAssembly allow us to maintain common patterns (for example, a chain of responsibility), and overcome some of the security risks.
“If we’d had WebAssembly at the time, we would undoubtedly have used it,” he said.
As the ”State of WebAssembly 2023” report suggested, this technology fits well when you start thinking about injectability, plugins and extensible systems. The tooling and the surrounding ecosystem continue to improve, the security story is strong, and the Wasm bytecode is compact, making it easy to pass around and share.
“We now believe that the user experience is to create a Metaresource and Policy Attachment, but if logic is needed, we can just create a WebAssembly binary,” Yacobucci said. “That way, you don’t have to restart your ingress controller or redeploy anything — although redeployment is an option if you want to keep your containers immutable.
“You can inject functionality and logic within your system, or remove it dynamically, without needing any interaction with your admin and security teams. Your security teams should feel pretty safe because WebAssembly is bastioned in a very strict way.”
Yacobucci wouldn’t comment on what the toolchain would look like, but he noted that F5 has recently acquired Suborbital, which has a very mature WebAssembly toolchain.
Another appealing quality of WebAssembly is that it offers a broad choice of programming languages. “It makes sense to write in Go for Kubernetes,” said Yacobucci. “However, Go is a statically-compiled language, which makes it harder to get dynamic plugin-style architectures.”
Bilgin Ibryam, principal product manager at Diagrid and co-author of “Kubernetes Patterns,” agreed that WebAssembly was a natural fit, noting that the service meshes and Dapr all make use of the technology.
He did, however, warn against the risk of putting too much into the gateway. “It’s important that the logic attached to the gateway is minimal, so it doesn’t end up being what ThoughtWorks calls overly ambitious,” he told us.
Yacobucci’s ultimate goal is to reach a place where users emergently design whole systems around a core group of control planes. “We have extensible data planes and an extensible future-proofed API,” he said. “Let’s make sure we have extensible control planes that take advantage of all that power.”
The post Why WebAssembly Is a Good Fit for Extensible Control Planes appeared first on The New Stack.
In software architecture, there's a constant push-pull between highly opinionated systems and more flexible options. Wasm might help resolve that conflict.