github-mirrors/terraform-aws-eks
1
0
Fork 0
mirror of https://github.com/ysoftdevs/terraform-aws-eks.git synced 2026-01-16 08:37:18 +01:00
Code Issues Packages Projects Releases Wiki Activity

docs: Update documentation for 1.24 changes to containerd, ensure examples are functioning as intended (#2463)

Browse Source 
Resolves undefined
This commit is contained in:
Bryant Biggs
2023-02-09 13:03:01 -05:00
committed by GitHub
parent aaea28dfa0
commit 07653b6249
8 changed files with 26 additions and 128 deletions
Download Patch File Download Diff File Expand all files Collapse all files

32
docs/compute_resources.md
View File

@@ -36,28 +36,7 @@ Refer to the [EKS Managed Node Group documentation](https://docs.aws.amazon.com/
}
```
3. Users have limited support to extend the user data that is pre-pended to the user data provided by the AWS EKS Managed Node Group service:
```hcl
eks_managed_node_groups = {
prepend_userdata = {
# See issue https://github.com/awslabs/amazon-eks-ami/issues/844
pre_bootstrap_user_data = <<-EOT
#!/bin/bash
set -ex
cat <<-EOF > /etc/profile.d/bootstrap.sh
export CONTAINER_RUNTIME="containerd"
export USE_MAX_PODS=false
export KUBELET_EXTRA_ARGS="--max-pods=110"
EOF
# Source extra environment variables in bootstrap script
sed -i '/^set -o errexit/a\\nsource /etc/profile.d/bootstrap.sh' /etc/eks/bootstrap.sh
EOT
}
}
```
4. Bottlerocket OS is supported in a similar manner. However, note that the user data for Bottlerocket OS uses the TOML format:
3. Bottlerocket OS is supported in a similar manner. However, note that the user data for Bottlerocket OS uses the TOML format:
```hcl
eks_managed_node_groups = {
@@ -74,7 +53,7 @@ Refer to the [EKS Managed Node Group documentation](https://docs.aws.amazon.com/
}
```
5. When using a custom AMI, the AWS EKS Managed Node Group service will NOT inject the necessary bootstrap script into the supplied user data. Users can elect to provide their own user data to bootstrap and connect or opt in to use the module provided user data:
4. When using a custom AMI, the AWS EKS Managed Node Group service will NOT inject the necessary bootstrap script into the supplied user data. Users can elect to provide their own user data to bootstrap and connect or opt in to use the module provided user data:
```hcl
eks_managed_node_groups = {
@@ -86,11 +65,8 @@ Refer to the [EKS Managed Node Group documentation](https://docs.aws.amazon.com/
# Note: this assumes the AMI provided is an EKS optimized AMI derivative
enable_bootstrap_user_data = true
bootstrap_extra_args = "--container-runtime containerd --kubelet-extra-args '--max-pods=20'"
pre_bootstrap_user_data = <<-EOT
export CONTAINER_RUNTIME="containerd"
export USE_MAX_PODS=false
export FOO=bar
EOT
# Because we have full control over the user data supplied, we can also run additional
@@ -102,7 +78,7 @@ Refer to the [EKS Managed Node Group documentation](https://docs.aws.amazon.com/
}
```
6. There is similar support for Bottlerocket OS:
5. There is similar support for Bottlerocket OS:
```hcl
eks_managed_node_groups = {

5
docs/faq.md
View File

@@ -1,11 +1,16 @@
# Frequently Asked Questions
- [Setting `disk_size` or `remote_access` does not make any changes](https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/faq.md#Settings-disk_size-or-remote_access-does-not-make-any-changes)
- [I received an error: `expect exactly one securityGroup tagged with kubernetes.io/cluster/<NAME> ...`](https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/faq.md#i-received-an-error-expect-exactly-one-securitygroup-tagged-with-kubernetesioclustername-)
- [Why are nodes not being registered?](https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/faq.md#why-are-nodes-not-being-registered)
- [Why are there no changes when a node group's `desired_size` is modified?](https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/faq.md#why-are-there-no-changes-when-a-node-groups-desired_size-is-modified)
- [How can I deploy Windows based nodes?](https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/faq.md#how-can-i-deploy-windows-based-nodes)
- [How do I access compute resource attributes?](https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/faq.md#how-do-i-access-compute-resource-attributes)
### Setting `disk_size` or `remote_access` does not make any changes
`disk_size`, and `remote_access` can only be set when using the EKS managed node group default launch template. This module defaults to providing a custom launch template to allow for custom security groups, tag propagation, etc. If you wish to forgo the custom launch template route, you can set `use_custom_launch_template = false` and then you can set `disk_size` and `remote_access`.
### I received an error: `expect exactly one securityGroup tagged with kubernetes.io/cluster/<NAME> ...`
By default, EKS creates a cluster primary security group that is created outside of the module and the EKS service adds the tag `{ "kubernetes.io/cluster/<CLUSTER_NAME>" = "owned" }`. This on its own does not cause any conflicts for addons such as the AWS Load Balancer Controller until users decide to attach both the cluster primary security group and the shared node security group created by the module (by setting `attach_cluster_primary_security_group = true`). The issue is not with having multiple security groups in your account with this tag key:value combination, but having multiple security groups with this tag key:value combination attached to nodes in the same cluster. There are a few ways to resolve this depending on your use case/intentions:

19
docs/user_data.md
View File

@@ -48,25 +48,6 @@ When using an EKS managed node group, users have 2 primary routes for interactin
| ℹ️ When using bottlerocket as the desired platform, since the user data for bottlerocket is TOML, all configurations are merged in the one file supplied as user data. Therefore, `pre_bootstrap_user_data` and `post_bootstrap_user_data` are not valid since the bottlerocket OS handles when various settings are applied. If you wish to supply additional configuration settings when using bottlerocket, supply them via the `bootstrap_extra_args` variable. For the linux platform, `bootstrap_extra_args` are settings that will be supplied to the [AWS EKS Optimized AMI bootstrap script](https://github.com/awslabs/amazon-eks-ami/blob/master/files/bootstrap.sh#L14) such as kubelet extra args, etc. See the [bottlerocket GitHub repository documentation](https://github.com/bottlerocket-os/bottlerocket#description-of-settings) for more details on what settings can be supplied via the `bootstrap_extra_args` variable. |
| :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
#### ⚠️ Caveat
Since the EKS Managed Node Group service provides the necessary bootstrap user data to nodes (unless an `ami_id` is provided), users do not have direct access to settings/variables provided by the EKS optimized AMI [`bootstrap.sh` script](https://github.com/awslabs/amazon-eks-ami/blob/master/files/bootstrap.sh). Currently, users must employ work-arounds to influence the `bootstrap.sh` script. For example, to enable `containerd` on EKS Managed Node Groups, users can supply the following user data. You can learn more about this issue [here](https://github.com/awslabs/amazon-eks-ami/issues/844):
```hcl
# See issue https://github.com/awslabs/amazon-eks-ami/issues/844
pre_bootstrap_user_data = <<-EOT
#!/bin/bash
set -ex
cat <<-EOF > /etc/profile.d/bootstrap.sh
export CONTAINER_RUNTIME="containerd"
export USE_MAX_PODS=false
export KUBELET_EXTRA_ARGS="--max-pods=110"
EOF
# Source extra environment variables in bootstrap script
sed -i '/^set -o errexit/a\\nsource /etc/profile.d/bootstrap.sh' /etc/eks/bootstrap.sh
EOT
```
### Self Managed Node Group
Self managed node groups require users to provide the necessary bootstrap user data. Users can elect to use the user data template provided by the module for their platform/OS or provide their own user data template for rendering by the module.

30
examples/eks_managed_node_group/README.md
View File

@@ -12,34 +12,6 @@ Configuration in this directory creates an AWS EKS cluster with various EKS Mana
See the [AWS documentation](https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html) for further details.
## Container Runtime & User Data
When using the default AMI provided by the EKS Managed Node Group service (i.e. - not specifying a value for `ami_id`), users should be aware of the limitations of configuring the node bootstrap process via user data. Due to not having direct access to the bootstrap.sh script invocation and therefore its configuration flags (this is provided by the EKS Managed Node Group service in the node user data), a workaround for ensuring the appropriate configuration settings is shown below. The following example shows how to inject configuration variables ahead of the merged user data provided by the EKS Managed Node Group service as well as how to enable the containerd runtime using this approach. More details can be found [here](https://github.com/awslabs/amazon-eks-ami/issues/844).
```hcl
...
# Demo of containerd usage when not specifying a custom AMI ID
# (merged into user data before EKS MNG provided user data)
containerd = {
name = "containerd"
# See issue https://github.com/awslabs/amazon-eks-ami/issues/844
pre_bootstrap_user_data = <<-EOT
#!/bin/bash
set -ex
cat <<-EOF > /etc/profile.d/bootstrap.sh
export CONTAINER_RUNTIME="containerd"
export USE_MAX_PODS=false
export KUBELET_EXTRA_ARGS="--max-pods=110"
EOF
# Source extra environment variables in bootstrap script
sed -i '/^set -o errexit/a\\nsource /etc/profile.d/bootstrap.sh' /etc/eks/bootstrap.sh
sed -i 's/KUBELET_EXTRA_ARGS=$2/KUBELET_EXTRA_ARGS="$2 $KUBELET_EXTRA_ARGS"/' /etc/eks/bootstrap.sh
EOT
}
...
```
## Usage
To run this example you need to execute:
@@ -71,7 +43,7 @@ Note that this example may create resources which cost money. Run `terraform des
| Name | Source | Version |
|------|--------|---------|
| <a name="module_ebs_kms_key"></a> [ebs\_kms\_key](#module\_ebs\_kms\_key) | terraform-aws-modules/kms/aws | ~> 1.1 |
| <a name="module_ebs_kms_key"></a> [ebs\_kms\_key](#module\_ebs\_kms\_key) | terraform-aws-modules/kms/aws | ~> 1.5 |
| <a name="module_eks"></a> [eks](#module\_eks) | ../.. | n/a |
| <a name="module_key_pair"></a> [key\_pair](#module\_key\_pair) | terraform-aws-modules/key-pair/aws | ~> 2.0 |
| <a name="module_vpc"></a> [vpc](#module\_vpc) | terraform-aws-modules/vpc/aws | ~> 3.0 |

31
examples/eks_managed_node_group/main.tf
View File

@@ -181,25 +181,6 @@ module "eks" {
instance_types = ["t4g.medium"]
}
# Demo of containerd usage when not specifying a custom AMI ID
# (merged into user data before EKS MNG provided user data)
containerd = {
name = "containerd"
# See issue https://github.com/awslabs/amazon-eks-ami/issues/844
pre_bootstrap_user_data = <<-EOT
#!/bin/bash
set -ex
cat <<-EOF > /etc/profile.d/bootstrap.sh
export CONTAINER_RUNTIME="containerd"
export USE_MAX_PODS=false
export KUBELET_EXTRA_ARGS="--max-pods=110"
EOF
# Source extra environment variables in bootstrap script
sed -i '/^set -o errexit/a\\nsource /etc/profile.d/bootstrap.sh' /etc/eks/bootstrap.sh
EOT
}
# Complete
complete = {
name = "complete-eks-mng"
@@ -213,11 +194,9 @@ module "eks" {
ami_id = data.aws_ami.eks_default.image_id
enable_bootstrap_user_data = true
bootstrap_extra_args = "--container-runtime containerd --kubelet-extra-args '--max-pods=20'"
pre_bootstrap_user_data = <<-EOT
export CONTAINER_RUNTIME="containerd"
export USE_MAX_PODS=false
export FOO=bar
EOT
post_bootstrap_user_data = <<-EOT
@@ -259,7 +238,7 @@ module "eks" {
iops = 3000
throughput = 150
encrypted = true
kms_key_id = module.ebs_kms_key.key_id
kms_key_id = module.ebs_kms_key.key_arn
delete_on_termination = true
}
}
@@ -315,6 +294,7 @@ module "vpc" {
public_subnet_ipv6_prefixes = [0, 1, 2]
private_subnet_ipv6_prefixes = [3, 4, 5]
intra_subnet_ipv6_prefixes = [6, 7, 8]
enable_nat_gateway = true
single_nat_gateway = true
@@ -355,7 +335,7 @@ module "vpc_cni_irsa" {
module "ebs_kms_key" {
source = "terraform-aws-modules/kms/aws"
version = "~> 1.1"
version = "~> 1.5"
description = "Customer managed key to encrypt EKS managed node group volumes"
@@ -363,7 +343,8 @@ module "ebs_kms_key" {
key_administrators = [
data.aws_caller_identity.current.arn
]
key_service_users = [
key_service_roles_for_autoscaling = [
# required for the ASG to manage encrypted volumes for nodes
"arn:aws:iam::${data.aws_caller_identity.current.account_id}:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling",
# required for the cluster / persistentvolume-controller to create encrypted PVCs

3
examples/self_managed_node_group/README.md
View File

@@ -39,7 +39,7 @@ Note that this example may create resources which cost money. Run `terraform des
| Name | Source | Version |
|------|--------|---------|
| <a name="module_ebs_kms_key"></a> [ebs\_kms\_key](#module\_ebs\_kms\_key) | terraform-aws-modules/kms/aws | ~> 1.1 |
| <a name="module_ebs_kms_key"></a> [ebs\_kms\_key](#module\_ebs\_kms\_key) | terraform-aws-modules/kms/aws | ~> 1.5 |
| <a name="module_eks"></a> [eks](#module\_eks) | ../.. | n/a |
| <a name="module_key_pair"></a> [key\_pair](#module\_key\_pair) | terraform-aws-modules/key-pair/aws | ~> 2.0 |
| <a name="module_vpc"></a> [vpc](#module\_vpc) | terraform-aws-modules/vpc/aws | ~> 3.0 |
@@ -48,7 +48,6 @@ Note that this example may create resources which cost money. Run `terraform des
| Name | Type |
|------|------|
| [aws_ec2_capacity_reservation.targeted](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ec2_capacity_reservation) | resource |
| [aws_iam_policy.additional](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_policy) | resource |
| [aws_ami.eks_default](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ami) | data source |
| [aws_ami.eks_default_bottlerocket](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ami) | data source |

27
examples/self_managed_node_group/main.tf
View File

@@ -182,12 +182,10 @@ module "eks" {
max_size = 7
desired_size = 1
ami_id = data.aws_ami.eks_default.id
bootstrap_extra_args = "--kubelet-extra-args '--max-pods=110'"
ami_id = data.aws_ami.eks_default.id
pre_bootstrap_user_data = <<-EOT
export CONTAINER_RUNTIME="containerd"
export USE_MAX_PODS=false
export FOO=bar
EOT
post_bootstrap_user_data = <<-EOT
@@ -212,7 +210,7 @@ module "eks" {
iops = 3000
throughput = 150
encrypted = true
kms_key_id = module.ebs_kms_key.key_id
kms_key_id = module.ebs_kms_key.key_arn
delete_on_termination = true
}
}
@@ -225,12 +223,6 @@ module "eks" {
instance_metadata_tags = "disabled"
}
capacity_reservation_specification = {
capacity_reservation_target = {
capacity_reservation_id = aws_ec2_capacity_reservation.targeted.id
}
}
create_iam_role = true
iam_role_name = "self-managed-node-group-complete-example"
iam_role_use_name_prefix = false
@@ -325,7 +317,7 @@ module "key_pair" {
module "ebs_kms_key" {
source = "terraform-aws-modules/kms/aws"
version = "~> 1.1"
version = "~> 1.5"
description = "Customer managed key to encrypt EKS managed node group volumes"
@@ -333,7 +325,8 @@ module "ebs_kms_key" {
key_administrators = [
data.aws_caller_identity.current.arn
]
key_service_users = [
key_service_roles_for_autoscaling = [
# required for the ASG to manage encrypted volumes for nodes
"arn:aws:iam::${data.aws_caller_identity.current.account_id}:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling",
# required for the cluster / persistentvolume-controller to create encrypted PVCs
@@ -346,14 +339,6 @@ module "ebs_kms_key" {
tags = local.tags
}
resource "aws_ec2_capacity_reservation" "targeted" {
instance_type = "m6i.large"
instance_platform = "Linux/UNIX"
availability_zone = "${local.region}a"
instance_count = 1
instance_match_criteria = "targeted"
}
resource "aws_iam_policy" "additional" {
name = "${local.name}-additional"
description = "Example usage of node additional policy"

7
examples/user_data/main.tf
View File

@@ -19,7 +19,7 @@ module "eks_mng_linux_additional" {
source = "../../modules/_user_data"
pre_bootstrap_user_data = <<-EOT
export CONTAINER_RUNTIME="containerd"
export USE_MAX_PODS=false
EOT
}
@@ -34,11 +34,10 @@ module "eks_mng_linux_custom_ami" {
enable_bootstrap_user_data = true
pre_bootstrap_user_data = <<-EOT
export CONTAINER_RUNTIME="containerd"
export USE_MAX_PODS=false
export FOO=bar
EOT
bootstrap_extra_args = "--container-runtime containerd --kubelet-extra-args '--max-pods=20 --instance-type t3a.large'"
bootstrap_extra_args = "--kubelet-extra-args '--instance-type t3a.large'"
post_bootstrap_user_data = <<-EOT
echo "All done"