On March 1, Docker will start enforcing its latest rate limits for pulling images from Docker Hub. While the professional (paid) accounts will still enjoy generous downloads, personal accounts will be limited to 40 pulls an hour (10 if unauthenticated).
For those running a small, experimental cluster at home, these limits can be a “major pain point,” noted developer Philip Laine,
Laine is the author of Spegel, an open source project that brings peer-to-peer file sharing to the world of container registries.
Spegel can help the hobbyists keep under their allotment, as well as help larger organization deploy their Kubernetes workload clusters much more quickly.
When Kubernetes starts or recovers a node, each node must pull each copy of the workload image from a nearby registry, be it cloud, public or individually self-hosted.
Spegel sets up a distributed registry across each node, so each unique container can be downloaded once and then copied to other nodes.
Introducing the technology in a FOSDEM talk earlier this month, the Spegel creator posted benchmarks showing that image retrieval times can be improved by 82%.
This approach also speeds workload startup times and reduces network traffic.
Stateless Cluster Local OCI Registry
Cloud native current bottleneck is distributing the workloads to the runtime.
But operations can be thwarted by rate limits of the image registry, or come to a halt entirely should a registry go offline. Or images can disappear, such as what happened with Red Hat‘s Quay recently.
“There is no point of having a cluster that can scale to 10,000 nodes if you can’t pull down the image for the application you actually run,” Laine told the audience.
If the image extremely large, just the network travel time can slow things down, doubly so for rate limits: Docker Hub, for instance, limits download speeds to 100MB/s for images.
Like BitTorrent, but for Containers
Laine recalled a meetup presentation from 2018 or so from a system administrator about how his organization worked through a Docker Hub outage during a dramatic increase of its own usership. Operations could not scale up because it depended on several critical images from the temporarily-offline Hub.
The sysadmin’s solution was to ssh into an old node, then exporting the images and then basically continuously copy the images to the new nodes as they came on line.
“It worked, even if it was a very dirty solution,” Laine said.
The workaround sparked an idea with Laine: “Why aren’t we always doing this?”
Thus began work on Spegel, an OCI-complaint read-only registry.
The implementation turned out to be surprisingly easy, according to Laine, who was also one of the core developers of Flux GitOps tool.
An image is actually a collection of multiple components. At the top level is an index of all the other included components. It references, in a JSON format, all the other layers with their digest or the cryptographic hash of the content itself.
The Linux Foundation‘s Open Container Interface (OCI) specifies how containers are structured. The OCI distribution spec describes how clients pull images from a registry, which is done layer by layer.
Spegel allows Kubernetes nodes to pull images from each other. If none of the nodes have the image, the requests falls back to the original registry.
One advantage that OCI’s containerd format provides is that it stores uncompressed images layers on disk. The file names are the hashes.
“This means Spegel can piggyback off of Containerd. It is not doing any type of storage. So the role of the registry is to serve as proxy. “There is no state at all,” he said.
Each node has an OCI registry, which looks first on the local host for the layer. If one is not found, Spegel intercepts the request and looks on other nodes for the requested layer. If one is found, it proxies the request.
Speqel has three components: the registry, a routing and discovery component, and an advertising mechanism.
“It’s a bit like BitTorrent,” he said. “It’s a bunch of clients advertising to other clients.”
The registry looks at all the content on disk and then advertises it in a distributed hash table. The software uses a Go implementation of the widely used Kademlia Distributed Hash Table. When a request comes in, Spegel simply looks it up on the hash table.
New Docker Rate Limits
Today, the majority of users of Spegel tend to be home lab enthusiasts who are hoping to avoid Docker Hub rate limiting, Laine said. Other uses can be in air-gapped deployments or running very large machine learning models, which would copied much more quickly if done locally.
And when the new rate limits of Docker Hub, even those with professional accounts might want to consider caching.
New Docker Hub rate limits as of March 1.
Spegel’s Compatibility with cloud providers varies according to their adherence to the OCI specs. Spgel works great with the Amazon Kubernetes Service, somewhat with the Azure Kubernetes Service and thus far, not at all with the Google Kubernetes Engine.
It also works great with MiniKube. It’s actually embedded in SUSE’s K3s and RKE2 Kubernetes distributions.
Ultimately, Spegel may help us rethink how image distribution could work through the power of peer-to-peer sharing, which would benefit the hobbyist and largest at-scale Kubernetes users alike.
TNS Analyst Lawrence Hecht contributed to this post.
The post Bypass Docker Hub Rate Limits With This Stateless Image Cache appeared first on The New Stack.
Frustrated by image registry rate limits, outages, and limited bandwidth? With the open source Spegel, Kubernetes nodes can pull images from each other, peer-to-peer style.