diff --git a/docs/compute_resources.md b/docs/compute_resources.md index a7138e8..5e1d59f 100644 --- a/docs/compute_resources.md +++ b/docs/compute_resources.md @@ -36,28 +36,7 @@ Refer to the [EKS Managed Node Group documentation](https://docs.aws.amazon.com/ } ``` -3. Users have limited support to extend the user data that is pre-pended to the user data provided by the AWS EKS Managed Node Group service: - -```hcl - eks_managed_node_groups = { - prepend_userdata = { - # See issue https://github.com/awslabs/amazon-eks-ami/issues/844 - pre_bootstrap_user_data = <<-EOT - #!/bin/bash - set -ex - cat <<-EOF > /etc/profile.d/bootstrap.sh - export CONTAINER_RUNTIME="containerd" - export USE_MAX_PODS=false - export KUBELET_EXTRA_ARGS="--max-pods=110" - EOF - # Source extra environment variables in bootstrap script - sed -i '/^set -o errexit/a\\nsource /etc/profile.d/bootstrap.sh' /etc/eks/bootstrap.sh - EOT - } - } -``` - -4. Bottlerocket OS is supported in a similar manner. However, note that the user data for Bottlerocket OS uses the TOML format: +3. Bottlerocket OS is supported in a similar manner. However, note that the user data for Bottlerocket OS uses the TOML format: ```hcl eks_managed_node_groups = { @@ -74,7 +53,7 @@ Refer to the [EKS Managed Node Group documentation](https://docs.aws.amazon.com/ } ``` -5. When using a custom AMI, the AWS EKS Managed Node Group service will NOT inject the necessary bootstrap script into the supplied user data. Users can elect to provide their own user data to bootstrap and connect or opt in to use the module provided user data: +4. When using a custom AMI, the AWS EKS Managed Node Group service will NOT inject the necessary bootstrap script into the supplied user data. Users can elect to provide their own user data to bootstrap and connect or opt in to use the module provided user data: ```hcl eks_managed_node_groups = { @@ -86,11 +65,8 @@ Refer to the [EKS Managed Node Group documentation](https://docs.aws.amazon.com/ # Note: this assumes the AMI provided is an EKS optimized AMI derivative enable_bootstrap_user_data = true - bootstrap_extra_args = "--container-runtime containerd --kubelet-extra-args '--max-pods=20'" - pre_bootstrap_user_data = <<-EOT - export CONTAINER_RUNTIME="containerd" - export USE_MAX_PODS=false + export FOO=bar EOT # Because we have full control over the user data supplied, we can also run additional @@ -102,7 +78,7 @@ Refer to the [EKS Managed Node Group documentation](https://docs.aws.amazon.com/ } ``` -6. There is similar support for Bottlerocket OS: +5. There is similar support for Bottlerocket OS: ```hcl eks_managed_node_groups = { diff --git a/docs/faq.md b/docs/faq.md index c53b027..215cdb2 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -1,11 +1,16 @@ # Frequently Asked Questions +- [Setting `disk_size` or `remote_access` does not make any changes](https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/faq.md#Settings-disk_size-or-remote_access-does-not-make-any-changes) - [I received an error: `expect exactly one securityGroup tagged with kubernetes.io/cluster/ ...`](https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/faq.md#i-received-an-error-expect-exactly-one-securitygroup-tagged-with-kubernetesioclustername-) - [Why are nodes not being registered?](https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/faq.md#why-are-nodes-not-being-registered) - [Why are there no changes when a node group's `desired_size` is modified?](https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/faq.md#why-are-there-no-changes-when-a-node-groups-desired_size-is-modified) - [How can I deploy Windows based nodes?](https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/faq.md#how-can-i-deploy-windows-based-nodes) - [How do I access compute resource attributes?](https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/faq.md#how-do-i-access-compute-resource-attributes) +### Setting `disk_size` or `remote_access` does not make any changes + +`disk_size`, and `remote_access` can only be set when using the EKS managed node group default launch template. This module defaults to providing a custom launch template to allow for custom security groups, tag propagation, etc. If you wish to forgo the custom launch template route, you can set `use_custom_launch_template = false` and then you can set `disk_size` and `remote_access`. + ### I received an error: `expect exactly one securityGroup tagged with kubernetes.io/cluster/ ...` By default, EKS creates a cluster primary security group that is created outside of the module and the EKS service adds the tag `{ "kubernetes.io/cluster/" = "owned" }`. This on its own does not cause any conflicts for addons such as the AWS Load Balancer Controller until users decide to attach both the cluster primary security group and the shared node security group created by the module (by setting `attach_cluster_primary_security_group = true`). The issue is not with having multiple security groups in your account with this tag key:value combination, but having multiple security groups with this tag key:value combination attached to nodes in the same cluster. There are a few ways to resolve this depending on your use case/intentions: diff --git a/docs/user_data.md b/docs/user_data.md index 4ce9a13..7b23694 100644 --- a/docs/user_data.md +++ b/docs/user_data.md @@ -48,25 +48,6 @@ When using an EKS managed node group, users have 2 primary routes for interactin | ℹ️ When using bottlerocket as the desired platform, since the user data for bottlerocket is TOML, all configurations are merged in the one file supplied as user data. Therefore, `pre_bootstrap_user_data` and `post_bootstrap_user_data` are not valid since the bottlerocket OS handles when various settings are applied. If you wish to supply additional configuration settings when using bottlerocket, supply them via the `bootstrap_extra_args` variable. For the linux platform, `bootstrap_extra_args` are settings that will be supplied to the [AWS EKS Optimized AMI bootstrap script](https://github.com/awslabs/amazon-eks-ami/blob/master/files/bootstrap.sh#L14) such as kubelet extra args, etc. See the [bottlerocket GitHub repository documentation](https://github.com/bottlerocket-os/bottlerocket#description-of-settings) for more details on what settings can be supplied via the `bootstrap_extra_args` variable. | | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -#### ⚠️ Caveat - -Since the EKS Managed Node Group service provides the necessary bootstrap user data to nodes (unless an `ami_id` is provided), users do not have direct access to settings/variables provided by the EKS optimized AMI [`bootstrap.sh` script](https://github.com/awslabs/amazon-eks-ami/blob/master/files/bootstrap.sh). Currently, users must employ work-arounds to influence the `bootstrap.sh` script. For example, to enable `containerd` on EKS Managed Node Groups, users can supply the following user data. You can learn more about this issue [here](https://github.com/awslabs/amazon-eks-ami/issues/844): - -```hcl - # See issue https://github.com/awslabs/amazon-eks-ami/issues/844 - pre_bootstrap_user_data = <<-EOT - #!/bin/bash - set -ex - cat <<-EOF > /etc/profile.d/bootstrap.sh - export CONTAINER_RUNTIME="containerd" - export USE_MAX_PODS=false - export KUBELET_EXTRA_ARGS="--max-pods=110" - EOF - # Source extra environment variables in bootstrap script - sed -i '/^set -o errexit/a\\nsource /etc/profile.d/bootstrap.sh' /etc/eks/bootstrap.sh - EOT -``` - ### Self Managed Node Group Self managed node groups require users to provide the necessary bootstrap user data. Users can elect to use the user data template provided by the module for their platform/OS or provide their own user data template for rendering by the module. diff --git a/examples/eks_managed_node_group/README.md b/examples/eks_managed_node_group/README.md index 45fa191..9328a05 100644 --- a/examples/eks_managed_node_group/README.md +++ b/examples/eks_managed_node_group/README.md @@ -12,34 +12,6 @@ Configuration in this directory creates an AWS EKS cluster with various EKS Mana See the [AWS documentation](https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html) for further details. -## Container Runtime & User Data - -When using the default AMI provided by the EKS Managed Node Group service (i.e. - not specifying a value for `ami_id`), users should be aware of the limitations of configuring the node bootstrap process via user data. Due to not having direct access to the bootstrap.sh script invocation and therefore its configuration flags (this is provided by the EKS Managed Node Group service in the node user data), a workaround for ensuring the appropriate configuration settings is shown below. The following example shows how to inject configuration variables ahead of the merged user data provided by the EKS Managed Node Group service as well as how to enable the containerd runtime using this approach. More details can be found [here](https://github.com/awslabs/amazon-eks-ami/issues/844). - -```hcl - ... - # Demo of containerd usage when not specifying a custom AMI ID - # (merged into user data before EKS MNG provided user data) - containerd = { - name = "containerd" - - # See issue https://github.com/awslabs/amazon-eks-ami/issues/844 - pre_bootstrap_user_data = <<-EOT - #!/bin/bash - set -ex - cat <<-EOF > /etc/profile.d/bootstrap.sh - export CONTAINER_RUNTIME="containerd" - export USE_MAX_PODS=false - export KUBELET_EXTRA_ARGS="--max-pods=110" - EOF - # Source extra environment variables in bootstrap script - sed -i '/^set -o errexit/a\\nsource /etc/profile.d/bootstrap.sh' /etc/eks/bootstrap.sh - sed -i 's/KUBELET_EXTRA_ARGS=$2/KUBELET_EXTRA_ARGS="$2 $KUBELET_EXTRA_ARGS"/' /etc/eks/bootstrap.sh - EOT - } - ... -``` - ## Usage To run this example you need to execute: @@ -71,7 +43,7 @@ Note that this example may create resources which cost money. Run `terraform des | Name | Source | Version | |------|--------|---------| -| [ebs\_kms\_key](#module\_ebs\_kms\_key) | terraform-aws-modules/kms/aws | ~> 1.1 | +| [ebs\_kms\_key](#module\_ebs\_kms\_key) | terraform-aws-modules/kms/aws | ~> 1.5 | | [eks](#module\_eks) | ../.. | n/a | | [key\_pair](#module\_key\_pair) | terraform-aws-modules/key-pair/aws | ~> 2.0 | | [vpc](#module\_vpc) | terraform-aws-modules/vpc/aws | ~> 3.0 | diff --git a/examples/eks_managed_node_group/main.tf b/examples/eks_managed_node_group/main.tf index acdd39a..2373d88 100644 --- a/examples/eks_managed_node_group/main.tf +++ b/examples/eks_managed_node_group/main.tf @@ -181,25 +181,6 @@ module "eks" { instance_types = ["t4g.medium"] } - # Demo of containerd usage when not specifying a custom AMI ID - # (merged into user data before EKS MNG provided user data) - containerd = { - name = "containerd" - - # See issue https://github.com/awslabs/amazon-eks-ami/issues/844 - pre_bootstrap_user_data = <<-EOT - #!/bin/bash - set -ex - cat <<-EOF > /etc/profile.d/bootstrap.sh - export CONTAINER_RUNTIME="containerd" - export USE_MAX_PODS=false - export KUBELET_EXTRA_ARGS="--max-pods=110" - EOF - # Source extra environment variables in bootstrap script - sed -i '/^set -o errexit/a\\nsource /etc/profile.d/bootstrap.sh' /etc/eks/bootstrap.sh - EOT - } - # Complete complete = { name = "complete-eks-mng" @@ -213,11 +194,9 @@ module "eks" { ami_id = data.aws_ami.eks_default.image_id enable_bootstrap_user_data = true - bootstrap_extra_args = "--container-runtime containerd --kubelet-extra-args '--max-pods=20'" pre_bootstrap_user_data = <<-EOT - export CONTAINER_RUNTIME="containerd" - export USE_MAX_PODS=false + export FOO=bar EOT post_bootstrap_user_data = <<-EOT @@ -259,7 +238,7 @@ module "eks" { iops = 3000 throughput = 150 encrypted = true - kms_key_id = module.ebs_kms_key.key_id + kms_key_id = module.ebs_kms_key.key_arn delete_on_termination = true } } @@ -315,6 +294,7 @@ module "vpc" { public_subnet_ipv6_prefixes = [0, 1, 2] private_subnet_ipv6_prefixes = [3, 4, 5] + intra_subnet_ipv6_prefixes = [6, 7, 8] enable_nat_gateway = true single_nat_gateway = true @@ -355,7 +335,7 @@ module "vpc_cni_irsa" { module "ebs_kms_key" { source = "terraform-aws-modules/kms/aws" - version = "~> 1.1" + version = "~> 1.5" description = "Customer managed key to encrypt EKS managed node group volumes" @@ -363,7 +343,8 @@ module "ebs_kms_key" { key_administrators = [ data.aws_caller_identity.current.arn ] - key_service_users = [ + + key_service_roles_for_autoscaling = [ # required for the ASG to manage encrypted volumes for nodes "arn:aws:iam::${data.aws_caller_identity.current.account_id}:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling", # required for the cluster / persistentvolume-controller to create encrypted PVCs diff --git a/examples/self_managed_node_group/README.md b/examples/self_managed_node_group/README.md index 9d78d22..d6944ac 100644 --- a/examples/self_managed_node_group/README.md +++ b/examples/self_managed_node_group/README.md @@ -39,7 +39,7 @@ Note that this example may create resources which cost money. Run `terraform des | Name | Source | Version | |------|--------|---------| -| [ebs\_kms\_key](#module\_ebs\_kms\_key) | terraform-aws-modules/kms/aws | ~> 1.1 | +| [ebs\_kms\_key](#module\_ebs\_kms\_key) | terraform-aws-modules/kms/aws | ~> 1.5 | | [eks](#module\_eks) | ../.. | n/a | | [key\_pair](#module\_key\_pair) | terraform-aws-modules/key-pair/aws | ~> 2.0 | | [vpc](#module\_vpc) | terraform-aws-modules/vpc/aws | ~> 3.0 | @@ -48,7 +48,6 @@ Note that this example may create resources which cost money. Run `terraform des | Name | Type | |------|------| -| [aws_ec2_capacity_reservation.targeted](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ec2_capacity_reservation) | resource | | [aws_iam_policy.additional](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_policy) | resource | | [aws_ami.eks_default](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ami) | data source | | [aws_ami.eks_default_bottlerocket](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ami) | data source | diff --git a/examples/self_managed_node_group/main.tf b/examples/self_managed_node_group/main.tf index fe7b5d8..7f1993d 100644 --- a/examples/self_managed_node_group/main.tf +++ b/examples/self_managed_node_group/main.tf @@ -182,12 +182,10 @@ module "eks" { max_size = 7 desired_size = 1 - ami_id = data.aws_ami.eks_default.id - bootstrap_extra_args = "--kubelet-extra-args '--max-pods=110'" + ami_id = data.aws_ami.eks_default.id pre_bootstrap_user_data = <<-EOT - export CONTAINER_RUNTIME="containerd" - export USE_MAX_PODS=false + export FOO=bar EOT post_bootstrap_user_data = <<-EOT @@ -212,7 +210,7 @@ module "eks" { iops = 3000 throughput = 150 encrypted = true - kms_key_id = module.ebs_kms_key.key_id + kms_key_id = module.ebs_kms_key.key_arn delete_on_termination = true } } @@ -225,12 +223,6 @@ module "eks" { instance_metadata_tags = "disabled" } - capacity_reservation_specification = { - capacity_reservation_target = { - capacity_reservation_id = aws_ec2_capacity_reservation.targeted.id - } - } - create_iam_role = true iam_role_name = "self-managed-node-group-complete-example" iam_role_use_name_prefix = false @@ -325,7 +317,7 @@ module "key_pair" { module "ebs_kms_key" { source = "terraform-aws-modules/kms/aws" - version = "~> 1.1" + version = "~> 1.5" description = "Customer managed key to encrypt EKS managed node group volumes" @@ -333,7 +325,8 @@ module "ebs_kms_key" { key_administrators = [ data.aws_caller_identity.current.arn ] - key_service_users = [ + + key_service_roles_for_autoscaling = [ # required for the ASG to manage encrypted volumes for nodes "arn:aws:iam::${data.aws_caller_identity.current.account_id}:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling", # required for the cluster / persistentvolume-controller to create encrypted PVCs @@ -346,14 +339,6 @@ module "ebs_kms_key" { tags = local.tags } -resource "aws_ec2_capacity_reservation" "targeted" { - instance_type = "m6i.large" - instance_platform = "Linux/UNIX" - availability_zone = "${local.region}a" - instance_count = 1 - instance_match_criteria = "targeted" -} - resource "aws_iam_policy" "additional" { name = "${local.name}-additional" description = "Example usage of node additional policy" diff --git a/examples/user_data/main.tf b/examples/user_data/main.tf index cb565ba..d7d5131 100644 --- a/examples/user_data/main.tf +++ b/examples/user_data/main.tf @@ -19,7 +19,7 @@ module "eks_mng_linux_additional" { source = "../../modules/_user_data" pre_bootstrap_user_data = <<-EOT - export CONTAINER_RUNTIME="containerd" + export USE_MAX_PODS=false EOT } @@ -34,11 +34,10 @@ module "eks_mng_linux_custom_ami" { enable_bootstrap_user_data = true pre_bootstrap_user_data = <<-EOT - export CONTAINER_RUNTIME="containerd" - export USE_MAX_PODS=false + export FOO=bar EOT - bootstrap_extra_args = "--container-runtime containerd --kubelet-extra-args '--max-pods=20 --instance-type t3a.large'" + bootstrap_extra_args = "--kubelet-extra-args '--instance-type t3a.large'" post_bootstrap_user_data = <<-EOT echo "All done"