This is the second part of a two-part article exploring Amazon Elastic Kubernetes Service (Amazon EKS) Auto Mode. In Part 1, we covered the core concepts of EKS Auto Mode, an overview of Karpenter, and the differences between EKS with Fargate and EKS Auto. In this part, we will see how to launch an EKS Auto cluster and delve into the internals.
EKS Auto vs. Traditional EKS
We can use eksctl to launch an EKS Auto cluster. All we have to do is pass the --enable-auto-mode switch.
The below command launches a cluster called tns-demo-auto:
eksctl create cluster --name=tns-demo-auto --enable-auto-mode
After waiting for a few minutes, the cluster will be ready. In parallel, I have launched a traditional EKS cluster with SSH access enabled. This cluster, which has a single node, is called tns-demo.
eksctl create cluster --name tns-demo --nodes 1 --ssh-access
When we look at the AWS Console, we can see both the clusters.
Switching to the EC2 Console, we can see two instances: One associated with the EKS Auto cluster and the other with the instance name belonging to the traditional cluster.
When we deploy a workload on the EKS Auto cluster and the pod goes into a pending state due to a lack of resources, EKS Auto will add a new node. Similarly, when the workload is deleted, and the node is idle, it automatically scales down the cluster.
The key difference between the EC2 instances associated with the two cluster types is that the traditional nodes have an IPV4 address assigned to them, enabling us to SSH into it.
The screenshot below shows the IP address associated with the EC2 instance that is part of the traditional EKS cluster.
When we SSH into it, we can see that the AMI runs Amazon Linux 2 and Kubernetes components, such as the kubelet.
The nodes belonging to EKS Auto Nodegroup don’t get an IP address. They are completely managed by the control plane, and you cannot SSH them. These instances are called Managed EC2 instances. The below screenshot shows the details of EC2 instances that’s a part of the EKS Auto cluster.
Understanding Managed EC2 Instances
In EKS Auto Mode, AWS is responsible for provisioning and managing the EC2 instances that serve as worker nodes in your Kubernetes cluster. These instances, however, are not your typical EC2 instances. They are managed instances with specific characteristics and limitations designed to streamline operations and enhance security.
Below are the key attributes of managed EC2 instances in EKS Auto Mode:
- AWS-managed AMIs: EKS Auto Mode uses a locked-down, Bottlerocket-based AMI that is preconfigured with essential Kubernetes components and security settings. Users cannot specify or customize the AMI.
- AMI and OS: Auto Mode uses a locked-down, Bottlerocket-based AMI with preconfigured systemd services for essential Kubernetes components. Users cannot specify or customize the AMI.
- Restricted Access: Direct access to worker nodes is restricted, with no SSH or SSM access allowed. This enhances security by limiting potential attack vectors.
- Automated Lifecycle: Nodes are automatically rotated every 21 days to ensure they are running the latest security updates.
- Resource Limits: EKS Auto Mode imposes a limit of 110 pods per node, which can affect node count and cost. This approach simplifies node management by offloading key tasks to AWS, but it also introduces some limitations in terms of customization and control.
In EKS Auto Mode, core add-ons like CoreDNS, kube-proxy and VPC CNI are run as systemd services within the AMI. This means they are not visible as pods within the cluster, which can make troubleshooting and customization more challenging. The Node Monitor Agent and Pod Identity Agent are also installed by default as systemd services.
The table below compares EC2 instances provisioned by a traditional EKS nodepool and EKS Auto:
Amazon EKS Auto vs. GKE Autopilot
GKE Autopilot is Google Cloud’s offering for a fully managed Kubernetes experience. Both EKS Auto Mode and GKE Autopilot aim to simplify cluster management, but they have some key differences:
- Node Management: GKE Autopilot manages the entire node life cycle, including provisioning, scaling and maintenance, while EKS Auto Mode provides more flexibility in terms of instance type selection and customization.
- Pricing: GKE Autopilot has a flat fee per cluster per hour plus workload costs, while EKS Auto Mode charges a 12% management fee on top of EC2 instance costs.
- Customization: Both offer limited customization options, but EKS Auto Mode provides more control over node configurations and add-on management.
- Security: GKE Autopilot uses a hardened default configuration with automatic security patches, while EKS Auto Mode relies on immutable AMIs and automatic node rotation.
- Networking: GKE Autopilot enforces network policies with GKE Dataplane V2, while EKS Auto Mode has limitations in terms of network customization.
- Upgrades: GKE Autopilot provides automatic upgrades with maintenance windows and exclusions, while EKS Auto Mode automates node version updates but requires manual intervention for control plane upgrades.
Amazon EKS Auto vs. AKS Automatic
AKS Automatic is Microsoft Azure’s solution for simplified Kubernetes management. Here’s how it compares to EKS Auto Mode:
- Automation Scope: AKS Automatic automates a wider range of tasks, including node and pod scaling, networking and security, while EKS Auto Mode focuses primarily on node management and add-on automation.
- Workload Scaling: AKS Automatic combines Karpenter’s node scaling with Azure’s workload autoscaling, providing a more comprehensive scaling solution.
- Security and Governance: AKS Automatic integrates with Azure Security Center and Entra ID for advanced security and governance capabilities, while EKS Auto Mode relies on strong defaults and automated node rotation.
- Networking: AKS Automatic automates VNet configuration and supports advanced networking setups, while EKS Auto Mode requires manual configuration of VPCs and subnets.
- Updates: AKS Automatic provides automated updates for both the control plane and nodepools, while EKS Auto Mode automates node version updates but requires manual intervention for control plane upgrades.
- Operational Burden: AKS Automatic significantly reduces operational overhead by automating the entire Kubernetes life cycle, while EKS Auto Mode still requires manual intervention for certain tasks.
Side-by-Side Comparison of EKS Auto, GKE Autopilot and AKS Automatic
EKS Auto Mode shares similarities with GKE Autopilot and AKS Automatic, both of which are managed Kubernetes offerings from Google Cloud Platform and Microsoft Azure, respectively. However, there are some key differences:
GKE Autopilot emphasizes workload isolation and pod-level billing, while AKS Automatic focuses on a streamlined setup with automated upgrades and a multitenant control plane. EKS Auto Mode, on the other hand, offers a balance between automation and customization, allowing users to leverage AWS’s expertise while retaining some control over their infrastructure.
Limitations of Amazon EKS Auto
While EKS Auto Mode offers significant advantages, there are some limitations and caveats to consider:
- Limited Customization: EKS Auto Mode provides limited customization options for node configurations and add-on management. Users cannot SSH into worker nodes or directly modify systemd services for add-ons.
- Cost Implication: EKS Auto Mode incurs a 12% management fee on top of EC2 instance costs, which can be a significant factor for cost-conscious users.
- Pod Limits: EKS Auto Mode imposes a limit of 110 pods per node, which can lead to an increased node count compared to self-managed Karpenter, potentially impacting costs.
- Debugging Challenges: The lack of direct node access can make debugging issues more challenging.
- Potential for Version Mismatches: Automated updates for Kubernetes components and the OS can potentially lead to version mismatches and downtime for stateful workloads.
EKS Auto Mode is now accessible in all commercial AWS regions, excluding China. Users can manage EKS Auto Mode clusters using different tools such as eksctl, AWS CLI, the AWS Management Console and their chosen Infrastructure as Code (IaC) solutions setups.
The post A Deep Dive Into Amazon EKS Auto (Part 2) appeared first on The New Stack.
In the second part of this series, we'll delve into the internals of the Amazon Elastic Kubernetes Service's Auto Mode cluster.