feat!: Replace the use of aws-auth configmap with EKS cluster access entry (#2858)

* feat: Replace `resolve_conflicts` with `resolve_conflicts_on_create`/`delete`; raise MSV of AWS provider to `v5.0` to support * fix: Replace dynamic DNS suffix for `sts:AssumeRole` API calls for static suffix * feat: Add module tag * feat: Align Karpenter permissions with Karpenter v1beta1/v0.32 permissions from upstream * refactor: Move `aws-auth` ConfigMap functionality to its own sub-module * chore: Update examples * feat: Add state `moved` block for Karpenter Pod Identity role re-name * fix: Correct variable `create` description * feat: Add support for cluster access entries * chore: Bump MSV of Terraform to `1.3` * fix: Replace defunct kubectl provider with an updated forked equivalent * chore: Update and validate examples for access entry; clean up provider usage * docs: Correct double redundant variable descriptions * feat: Add support for Cloudwatch log group class argument * fix: Update usage tag placement, fix Karpenter event spelling, add upcoming changes section to upgrade guide * feat: Update Karpenter module to generalize naming used and align policy with the upstream Karpenter policy * feat: Add native support for Windows based managed nodegroups similar to AL2 and Bottlerocket * feat: Update self-managed nodegroup module to use latest features of ASG * docs: Update and simplify docs * fix: Correct variable description for AMI types * fix: Update upgrade guide with changes; rename Karpenter controller resource names to support migrating for users * docs: Complete upgrade guide docs for migration and changes applied * Update examples/karpenter/README.md Co-authored-by: Anton Babenko <anton@antonbabenko.com> * Update examples/outposts/README.md Co-authored-by: Anton Babenko <anton@antonbabenko.com> * Update modules/karpenter/README.md Co-authored-by: Anton Babenko <anton@antonbabenko.com> --------- Co-authored-by: Anton Babenko <anton@antonbabenko.com>
2026-03-19 16:21:09 +01:00 · 2024-02-02 09:36:25 -05:00
parent 2cb1fac31b
commit 6b40bdbb1d
71 changed files with 1809 additions and 2136 deletions
--- a/examples/karpenter/README.md
+++ b/examples/karpenter/README.md
@@ -41,6 +41,9 @@ kubectl delete node -l karpenter.sh/provisioner-name=default
 2. Remove the resources created by Terraform

 ```bash
+# Necessary to avoid removing Terraform's permissions too soon before its finished
+# cleaning up the resources it deployed inside the clsuter
+terraform state rm 'module.eks.aws_eks_access_entry.this["cluster_creator_admin"]' || true
 terraform destroy
 ```

@@ -51,21 +54,19 @@ Note that this example may create resources which cost money. Run `terraform des

 | Name | Version |
 |------|---------|
-| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0 |
-| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 4.57 |
+| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.3 |
+| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 5.34 |
 | <a name="requirement_helm"></a> [helm](#requirement\_helm) | >= 2.7 |
-| <a name="requirement_kubectl"></a> [kubectl](#requirement\_kubectl) | >= 1.14 |
-| <a name="requirement_kubernetes"></a> [kubernetes](#requirement\_kubernetes) | >= 2.10 |
-| <a name="requirement_null"></a> [null](#requirement\_null) | >= 3.0 |
+| <a name="requirement_kubectl"></a> [kubectl](#requirement\_kubectl) | >= 2.0 |

 ## Providers

 | Name | Version |
 |------|---------|
-| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 4.57 |
-| <a name="provider_aws.virginia"></a> [aws.virginia](#provider\_aws.virginia) | >= 4.57 |
+| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 5.34 |
+| <a name="provider_aws.virginia"></a> [aws.virginia](#provider\_aws.virginia) | >= 5.34 |
 | <a name="provider_helm"></a> [helm](#provider\_helm) | >= 2.7 |
-| <a name="provider_kubectl"></a> [kubectl](#provider\_kubectl) | >= 1.14 |
+| <a name="provider_kubectl"></a> [kubectl](#provider\_kubectl) | >= 2.0 |

 ## Modules

@@ -73,6 +74,7 @@ Note that this example may create resources which cost money. Run `terraform des
 |------|--------|---------|
 | <a name="module_eks"></a> [eks](#module\_eks) | ../.. | n/a |
 | <a name="module_karpenter"></a> [karpenter](#module\_karpenter) | ../../modules/karpenter | n/a |
+| <a name="module_karpenter_disabled"></a> [karpenter\_disabled](#module\_karpenter\_disabled) | ../../modules/karpenter | n/a |
 | <a name="module_vpc"></a> [vpc](#module\_vpc) | terraform-aws-modules/vpc/aws | ~> 5.0 |

 ## Resources
@@ -80,9 +82,9 @@ Note that this example may create resources which cost money. Run `terraform des
 | Name | Type |
 |------|------|
 | [helm_release.karpenter](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource |
-| [kubectl_manifest.karpenter_example_deployment](https://registry.terraform.io/providers/gavinbunney/kubectl/latest/docs/resources/manifest) | resource |
-| [kubectl_manifest.karpenter_node_class](https://registry.terraform.io/providers/gavinbunney/kubectl/latest/docs/resources/manifest) | resource |
-| [kubectl_manifest.karpenter_node_pool](https://registry.terraform.io/providers/gavinbunney/kubectl/latest/docs/resources/manifest) | resource |
+| [kubectl_manifest.karpenter_example_deployment](https://registry.terraform.io/providers/alekc/kubectl/latest/docs/resources/manifest) | resource |
+| [kubectl_manifest.karpenter_node_class](https://registry.terraform.io/providers/alekc/kubectl/latest/docs/resources/manifest) | resource |
+| [kubectl_manifest.karpenter_node_pool](https://registry.terraform.io/providers/alekc/kubectl/latest/docs/resources/manifest) | resource |
 | [aws_availability_zones.available](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/availability_zones) | data source |
 | [aws_ecrpublic_authorization_token.token](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ecrpublic_authorization_token) | data source |

@@ -94,7 +96,7 @@ No inputs.

 | Name | Description |
 |------|-------------|
-| <a name="output_aws_auth_configmap_yaml"></a> [aws\_auth\_configmap\_yaml](#output\_aws\_auth\_configmap\_yaml) | Formatted yaml output for base aws-auth configmap containing roles used in cluster node groups/fargate profiles |
+| <a name="output_access_entries"></a> [access\_entries](#output\_access\_entries) | Map of access entries created and their attributes |
 | <a name="output_cloudwatch_log_group_arn"></a> [cloudwatch\_log\_group\_arn](#output\_cloudwatch\_log\_group\_arn) | Arn of cloudwatch log group created |
 | <a name="output_cloudwatch_log_group_name"></a> [cloudwatch\_log\_group\_name](#output\_cloudwatch\_log\_group\_name) | Name of cloudwatch log group created |
 | <a name="output_cluster_addons"></a> [cluster\_addons](#output\_cluster\_addons) | Map of attribute maps for all EKS cluster addons enabled |
@@ -118,19 +120,19 @@ No inputs.
 | <a name="output_eks_managed_node_groups_autoscaling_group_names"></a> [eks\_managed\_node\_groups\_autoscaling\_group\_names](#output\_eks\_managed\_node\_groups\_autoscaling\_group\_names) | List of the autoscaling group names created by EKS managed node groups |
 | <a name="output_fargate_profiles"></a> [fargate\_profiles](#output\_fargate\_profiles) | Map of attribute maps for all EKS Fargate Profiles created |
 | <a name="output_karpenter_event_rules"></a> [karpenter\_event\_rules](#output\_karpenter\_event\_rules) | Map of the event rules created and their attributes |
+| <a name="output_karpenter_iam_role_arn"></a> [karpenter\_iam\_role\_arn](#output\_karpenter\_iam\_role\_arn) | The Amazon Resource Name (ARN) specifying the controller IAM role |
+| <a name="output_karpenter_iam_role_name"></a> [karpenter\_iam\_role\_name](#output\_karpenter\_iam\_role\_name) | The name of the controller IAM role |
+| <a name="output_karpenter_iam_role_unique_id"></a> [karpenter\_iam\_role\_unique\_id](#output\_karpenter\_iam\_role\_unique\_id) | Stable and unique string identifying the controller IAM role |
 | <a name="output_karpenter_instance_profile_arn"></a> [karpenter\_instance\_profile\_arn](#output\_karpenter\_instance\_profile\_arn) | ARN assigned by AWS to the instance profile |
 | <a name="output_karpenter_instance_profile_id"></a> [karpenter\_instance\_profile\_id](#output\_karpenter\_instance\_profile\_id) | Instance profile's ID |
 | <a name="output_karpenter_instance_profile_name"></a> [karpenter\_instance\_profile\_name](#output\_karpenter\_instance\_profile\_name) | Name of the instance profile |
 | <a name="output_karpenter_instance_profile_unique"></a> [karpenter\_instance\_profile\_unique](#output\_karpenter\_instance\_profile\_unique) | Stable and unique string identifying the IAM instance profile |
-| <a name="output_karpenter_irsa_arn"></a> [karpenter\_irsa\_arn](#output\_karpenter\_irsa\_arn) | The Amazon Resource Name (ARN) specifying the IAM role for service accounts |
-| <a name="output_karpenter_irsa_name"></a> [karpenter\_irsa\_name](#output\_karpenter\_irsa\_name) | The name of the IAM role for service accounts |
-| <a name="output_karpenter_irsa_unique_id"></a> [karpenter\_irsa\_unique\_id](#output\_karpenter\_irsa\_unique\_id) | Stable and unique string identifying the IAM role for service accounts |
+| <a name="output_karpenter_node_iam_role_arn"></a> [karpenter\_node\_iam\_role\_arn](#output\_karpenter\_node\_iam\_role\_arn) | The Amazon Resource Name (ARN) specifying the IAM role |
+| <a name="output_karpenter_node_iam_role_name"></a> [karpenter\_node\_iam\_role\_name](#output\_karpenter\_node\_iam\_role\_name) | The name of the IAM role |
+| <a name="output_karpenter_node_iam_role_unique_id"></a> [karpenter\_node\_iam\_role\_unique\_id](#output\_karpenter\_node\_iam\_role\_unique\_id) | Stable and unique string identifying the IAM role |
 | <a name="output_karpenter_queue_arn"></a> [karpenter\_queue\_arn](#output\_karpenter\_queue\_arn) | The ARN of the SQS queue |
 | <a name="output_karpenter_queue_name"></a> [karpenter\_queue\_name](#output\_karpenter\_queue\_name) | The name of the created Amazon SQS queue |
 | <a name="output_karpenter_queue_url"></a> [karpenter\_queue\_url](#output\_karpenter\_queue\_url) | The URL for the created Amazon SQS queue |
-| <a name="output_karpenter_role_arn"></a> [karpenter\_role\_arn](#output\_karpenter\_role\_arn) | The Amazon Resource Name (ARN) specifying the IAM role |
-| <a name="output_karpenter_role_name"></a> [karpenter\_role\_name](#output\_karpenter\_role\_name) | The name of the IAM role |
-| <a name="output_karpenter_role_unique_id"></a> [karpenter\_role\_unique\_id](#output\_karpenter\_role\_unique\_id) | Stable and unique string identifying the IAM role |
 | <a name="output_node_security_group_arn"></a> [node\_security\_group\_arn](#output\_node\_security\_group\_arn) | Amazon Resource Name (ARN) of the node shared security group |
 | <a name="output_node_security_group_id"></a> [node\_security\_group\_id](#output\_node\_security\_group\_id) | ID of the node shared security group |
 | <a name="output_oidc_provider"></a> [oidc\_provider](#output\_oidc\_provider) | The OpenID Connect identity provider (issuer URL without leading `https://`) |
--- a/examples/karpenter/main.tf
+++ b/examples/karpenter/main.tf
@@ -7,18 +7,6 @@ provider "aws" {
  alias  = "virginia"
 }

-provider "kubernetes" {
-  host                   = module.eks.cluster_endpoint
-  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
-
-  exec {
-    api_version = "client.authentication.k8s.io/v1beta1"
-    command     = "aws"
-    # This requires the awscli to be installed locally where Terraform is executed
-    args = ["eks", "get-token", "--cluster-name", module.eks.cluster_name]
-  }
-}
-
 provider "helm" {
  kubernetes {
    host                   = module.eks.cluster_endpoint
@@ -54,7 +42,7 @@ data "aws_ecrpublic_authorization_token" "token" {

 locals {
  name            = "ex-${replace(basename(path.cwd), "_", "-")}"
-  cluster_version = "1.27"
+  cluster_version = "1.28"
  region          = "eu-west-1"

  vpc_cidr = "10.0.0.0/16"
@@ -78,9 +66,11 @@ module "eks" {
  cluster_version                = local.cluster_version
  cluster_endpoint_public_access = true

+  # Gives Terraform identity admin access to cluster which will
+  # allow deploying resources (Karpenter) into the cluster
+  enable_cluster_creator_admin_permissions = true
+
  cluster_addons = {
-    kube-proxy = {}
-    vpc-cni    = {}
    coredns = {
      configuration_values = jsonencode({
        computeType = "Fargate"
@@ -106,6 +96,8 @@ module "eks" {
        }
      })
    }
+    kube-proxy = {}
+    vpc-cni    = {}
  }

  vpc_id                   = module.vpc.vpc_id
@@ -116,19 +108,6 @@ module "eks" {
  create_cluster_security_group = false
  create_node_security_group    = false

-  manage_aws_auth_configmap = true
-  aws_auth_roles = [
-    # We need to add in the Karpenter node IAM role for nodes launched by Karpenter
-    {
-      rolearn  = module.karpenter.role_arn
-      username = "system:node:{{EC2PrivateDNSName}}"
-      groups = [
-        "system:bootstrappers",
-        "system:nodes",
-      ]
-    },
-  ]
-
  fargate_profiles = {
    karpenter = {
      selectors = [
@@ -157,41 +136,51 @@ module "eks" {
 module "karpenter" {
  source = "../../modules/karpenter"

-  cluster_name           = module.eks.cluster_name
+  cluster_name = module.eks.cluster_name
+
+  # EKS Fargate currently does not support Pod Identity
+  enable_irsa            = true
  irsa_oidc_provider_arn = module.eks.oidc_provider_arn

-  # In v0.32.0/v1beta1, Karpenter now creates the IAM instance profile
-  # so we disable the Terraform creation and add the necessary permissions for Karpenter IRSA
-  enable_karpenter_instance_profile_creation = true
-
  # Used to attach additional IAM policies to the Karpenter node IAM role
-  iam_role_additional_policies = {
+  node_iam_role_additional_policies = {
    AmazonSSMManagedInstanceCore = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
  }

  tags = local.tags
 }

-resource "helm_release" "karpenter" {
-  namespace        = "karpenter"
-  create_namespace = true
+module "karpenter_disabled" {
+  source = "../../modules/karpenter"

+  create = false
+}
+
+################################################################################
+# Karpenter Helm chart & manifests
+# Not required; just to demonstrate functionality of the sub-module
+################################################################################
+
+resource "helm_release" "karpenter" {
+  namespace           = "karpenter"
+  create_namespace    = true
  name                = "karpenter"
  repository          = "oci://public.ecr.aws/karpenter"
  repository_username = data.aws_ecrpublic_authorization_token.token.user_name
  repository_password = data.aws_ecrpublic_authorization_token.token.password
  chart               = "karpenter"
-  version             = "v0.32.1"
+  version             = "v0.33.1"
+  wait                = false

  values = [
    <<-EOT
    settings:
      clusterName: ${module.eks.cluster_name}
      clusterEndpoint: ${module.eks.cluster_endpoint}
-      interruptionQueueName: ${module.karpenter.queue_name}
+      interruptionQueue: ${module.karpenter.queue_name}
    serviceAccount:
      annotations:
-        eks.amazonaws.com/role-arn: ${module.karpenter.irsa_arn} 
+        eks.amazonaws.com/role-arn: ${module.karpenter.iam_role_arn}
    EOT
  ]
 }
@@ -204,7 +193,7 @@ resource "kubectl_manifest" "karpenter_node_class" {
      name: default
    spec:
      amiFamily: AL2
-      role: ${module.karpenter.role_name}
+      role: ${module.karpenter.node_iam_role_name}
      subnetSelectorTerms:
        - tags:
            karpenter.sh/discovery: ${module.eks.cluster_name}
--- a/examples/karpenter/outputs.tf
+++ b/examples/karpenter/outputs.tf
@@ -47,6 +47,15 @@ output "cluster_primary_security_group_id" {
  value       = module.eks.cluster_primary_security_group_id
 }

+################################################################################
+# Access Entry
+################################################################################
+
+output "access_entries" {
+  description = "Map of access entries created and their attributes"
+  value       = module.eks.access_entries
+}
+
 ################################################################################
 # Security Group
 ################################################################################
@@ -183,31 +192,22 @@ output "self_managed_node_groups_autoscaling_group_names" {
 }

 ################################################################################
-# Additional
+# Karpenter controller IAM Role
 ################################################################################

-output "aws_auth_configmap_yaml" {
-  description = "Formatted yaml output for base aws-auth configmap containing roles used in cluster node groups/fargate profiles"
-  value       = module.eks.aws_auth_configmap_yaml
+output "karpenter_iam_role_name" {
+  description = "The name of the controller IAM role"
+  value       = module.karpenter.iam_role_name
 }

-################################################################################
-# IAM Role for Service Account (IRSA)
-################################################################################
-
-output "karpenter_irsa_name" {
-  description = "The name of the IAM role for service accounts"
-  value       = module.karpenter.irsa_name
+output "karpenter_iam_role_arn" {
+  description = "The Amazon Resource Name (ARN) specifying the controller IAM role"
+  value       = module.karpenter.iam_role_arn
 }

-output "karpenter_irsa_arn" {
-  description = "The Amazon Resource Name (ARN) specifying the IAM role for service accounts"
-  value       = module.karpenter.irsa_arn
-}
-
-output "karpenter_irsa_unique_id" {
-  description = "Stable and unique string identifying the IAM role for service accounts"
-  value       = module.karpenter.irsa_unique_id
+output "karpenter_iam_role_unique_id" {
+  description = "Stable and unique string identifying the controller IAM role"
+  value       = module.karpenter.iam_role_unique_id
 }

 ################################################################################
@@ -242,19 +242,19 @@ output "karpenter_event_rules" {
 # Node IAM Role
 ################################################################################

-output "karpenter_role_name" {
+output "karpenter_node_iam_role_name" {
  description = "The name of the IAM role"
-  value       = module.karpenter.role_name
+  value       = module.karpenter.node_iam_role_name
 }

-output "karpenter_role_arn" {
+output "karpenter_node_iam_role_arn" {
  description = "The Amazon Resource Name (ARN) specifying the IAM role"
-  value       = module.karpenter.role_arn
+  value       = module.karpenter.node_iam_role_arn
 }

-output "karpenter_role_unique_id" {
+output "karpenter_node_iam_role_unique_id" {
  description = "Stable and unique string identifying the IAM role"
-  value       = module.karpenter.role_unique_id
+  value       = module.karpenter.node_iam_role_unique_id
 }

 ################################################################################
--- a/examples/karpenter/versions.tf
+++ b/examples/karpenter/versions.tf
@@ -1,26 +1,18 @@
 terraform {
-  required_version = ">= 1.0"
+  required_version = ">= 1.3"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
-      version = ">= 4.57"
-    }
-    kubernetes = {
-      source  = "hashicorp/kubernetes"
-      version = ">= 2.10"
+      version = ">= 5.34"
    }
    helm = {
      source  = "hashicorp/helm"
      version = ">= 2.7"
    }
    kubectl = {
-      source  = "gavinbunney/kubectl"
-      version = ">= 1.14"
-    }
-    null = {
-      source  = "hashicorp/null"
-      version = ">= 3.0"
+      source  = "alekc/kubectl"
+      version = ">= 2.0"
    }
  }
 }