github-mirrors/terraform-aws-eks
1
0
Fork 0
mirror of https://github.com/ysoftdevs/terraform-aws-eks.git synced 2026-03-17 23:13:46 +01:00
Code Issues Packages Projects Releases Wiki Activity

feat!: Replace the use of aws-auth configmap with EKS cluster access entry (#2858)

Browse Source 
* feat: Replace `resolve_conflicts` with `resolve_conflicts_on_create`/`delete`; raise MSV of AWS provider to `v5.0` to support

* fix: Replace dynamic DNS suffix for `sts:AssumeRole` API calls for static suffix

* feat: Add module tag

* feat: Align Karpenter permissions with Karpenter v1beta1/v0.32 permissions from upstream

* refactor: Move `aws-auth` ConfigMap functionality to its own sub-module

* chore: Update examples

* feat: Add state `moved` block for Karpenter Pod Identity role re-name

* fix: Correct variable `create` description

* feat: Add support for cluster access entries

* chore: Bump MSV of Terraform to `1.3`

* fix: Replace defunct kubectl provider with an updated forked equivalent

* chore: Update and validate examples for access entry; clean up provider usage

* docs: Correct double redundant variable descriptions

* feat: Add support for Cloudwatch log group class argument

* fix: Update usage tag placement, fix Karpenter event spelling, add upcoming changes section to upgrade guide

* feat: Update Karpenter module to generalize naming used and align policy with the upstream Karpenter policy

* feat: Add native support for Windows based managed nodegroups similar to AL2 and Bottlerocket

* feat: Update self-managed nodegroup module to use latest features of ASG

* docs: Update and simplify docs

* fix: Correct variable description for AMI types

* fix: Update upgrade guide with changes; rename Karpenter controller resource names to support migrating for users

* docs: Complete upgrade guide docs for migration and changes applied

* Update examples/karpenter/README.md

Co-authored-by: Anton Babenko <anton@antonbabenko.com>

* Update examples/outposts/README.md

Co-authored-by: Anton Babenko <anton@antonbabenko.com>

* Update modules/karpenter/README.md

Co-authored-by: Anton Babenko <anton@antonbabenko.com>

---------

Co-authored-by: Anton Babenko <anton@antonbabenko.com>
This commit is contained in:
Bryant Biggs
2024-02-02 09:36:25 -05:00
committed by GitHub
parent 2cb1fac31b
commit 6b40bdbb1d
71 changed files with 1809 additions and 2136 deletions
Download Patch File Download Diff File Expand all files Collapse all files

145
modules/karpenter/README.md
View File

@@ -7,30 +7,15 @@ Configuration in this directory creates the AWS resources required by Karpenter
### All Resources (Default)
In the following example, the Karpenter module will create:
- An IAM role for service accounts (IRSA) with a narrowly scoped IAM policy for the Karpenter controller to utilize
- An IAM role and instance profile for the nodes created by Karpenter to utilize
- Note: This IAM role ARN will need to be added to the `aws-auth` configmap for nodes to join the cluster successfully
- An SQS queue and Eventbridge event rules for Karpenter to utilize for spot termination handling, capacity rebalancing, etc.
This setup is great for running Karpenter on EKS Fargate:
- An IAM role for use with Pod Identity and a scoped IAM policy for the Karpenter controller
- A Node IAM role that Karpenter will use to create an Instance Profile for the nodes to receive IAM permissions
- An access entry for the Node IAM role to allow nodes to join the cluster
- SQS queue and EventBridge event rules for Karpenter to utilize for spot termination handling, capacity re-balancing, etc.
```hcl
module "eks" {
source = "terraform-aws-modules/eks"
source = "terraform-aws-modules/eks/aws"
# Shown just for connection between cluster and Karpenter sub-module below
manage_aws_auth_configmap = true
aws_auth_roles = [
# We need to add in the Karpenter node IAM role for nodes launched by Karpenter
{
rolearn = module.karpenter.role_arn
username = "system:node:{{EC2PrivateDNSName}}"
groups = [
"system:bootstrappers",
"system:nodes",
]
},
]
...
}
@@ -39,11 +24,8 @@ module "karpenter" {
cluster_name = module.eks.cluster_name
irsa_oidc_provider_arn = module.eks.oidc_provider_arn
irsa_namespace_service_accounts = ["karpenter:karpenter"]
# Attach additional IAM policies to the Karpenter node IAM role
iam_role_additional_policies = {
node_iam_role_additional_policies = {
AmazonSSMManagedInstanceCore = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
}
@@ -54,15 +36,13 @@ module "karpenter" {
}
```
### External Node IAM Role (Default)
### Re-Use Existing Node IAM Role
In the following example, the Karpenter module will create:
- An IAM role for service accounts (IRSA) with a narrowly scoped IAM policy for the Karpenter controller to utilize
- An IAM instance profile for the nodes created by Karpenter to utilize
- Note: This setup will utilize the existing IAM role created by the EKS Managed Node group which means the role is already populated in the `aws-auth` configmap and no further updates are required.
- An SQS queue and Eventbridge event rules for Karpenter to utilize for spot termination handling, capacity rebalancing, etc.
- An IAM role for use with Pod Identity and a scoped IAM policy for the Karpenter controller
- SQS queue and EventBridge event rules for Karpenter to utilize for spot termination handling, capacity re-balancing, etc.
In this scenario, Karpenter would run atop the EKS Managed Node group and scale out nodes as needed from there:
In this scenario, Karpenter will re-use an existing Node IAM role from the EKS managed nodegroup which already has the necessary access entry permissions:
```hcl
module "eks" {
@@ -86,12 +66,12 @@ module "karpenter" {
cluster_name = module.eks.cluster_name
irsa_oidc_provider_arn = module.eks.oidc_provider_arn
irsa_namespace_service_accounts = ["karpenter:karpenter"]
create_iam_role = false
iam_role_arn = module.eks.eks_managed_node_groups["initial"].iam_role_arn
# Since the nodegroup role will already have an access entry
create_access_entry = false
tags = {
Environment = "dev"
Terraform = "true"
@@ -104,14 +84,14 @@ module "karpenter" {
| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 4.57 |
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.3 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 5.34 |
## Providers
| Name | Version |
|------|---------|
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 4.57 |
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 5.34 |
## Modules
@@ -123,61 +103,65 @@ No modules.
|------|------|
| [aws_cloudwatch_event_rule.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_event_rule) | resource |
| [aws_cloudwatch_event_target.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_event_target) | resource |
| [aws_eks_access_entry.node](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_access_entry) | resource |
| [aws_iam_instance_profile.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_instance_profile) | resource |
| [aws_iam_policy.irsa](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_policy) | resource |
| [aws_iam_role.irsa](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role) | resource |
| [aws_iam_role.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role) | resource |
| [aws_iam_role_policy_attachment.additional](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource |
| [aws_iam_role_policy_attachment.irsa](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource |
| [aws_iam_role_policy_attachment.irsa_additional](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource |
| [aws_iam_role_policy_attachment.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource |
| [aws_iam_policy.controller](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_policy) | resource |
| [aws_iam_role.controller](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role) | resource |
| [aws_iam_role.node](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role) | resource |
| [aws_iam_role_policy_attachment.controller](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource |
| [aws_iam_role_policy_attachment.controller_additional](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource |
| [aws_iam_role_policy_attachment.node](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource |
| [aws_iam_role_policy_attachment.node_additional](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource |
| [aws_sqs_queue.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sqs_queue) | resource |
| [aws_sqs_queue_policy.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sqs_queue_policy) | resource |
| [aws_caller_identity.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/caller_identity) | data source |
| [aws_iam_policy_document.assume_role](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
| [aws_iam_policy_document.irsa](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
| [aws_iam_policy_document.irsa_assume_role](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
| [aws_iam_policy_document.controller](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
| [aws_iam_policy_document.controller_assume_role](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
| [aws_iam_policy_document.node_assume_role](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
| [aws_iam_policy_document.queue](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
| [aws_partition.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/partition) | data source |
| [aws_region.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/region) | data source |
## Inputs
| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| <a name="input_cluster_ip_family"></a> [cluster\_ip\_family](#input\_cluster\_ip\_family) | The IP family used to assign Kubernetes pod and service addresses. Valid values are `ipv4` (default) and `ipv6` | `string` | `null` | no |
| <a name="input_access_entry_type"></a> [access\_entry\_type](#input\_access\_entry\_type) | Type of the access entry. `EC2_LINUX`, `FARGATE_LINUX`, or `EC2_WINDOWS`; defaults to `EC2_LINUX` | `string` | `"EC2_LINUX"` | no |
| <a name="input_ami_id_ssm_parameter_arns"></a> [ami\_id\_ssm\_parameter\_arns](#input\_ami\_id\_ssm\_parameter\_arns) | List of SSM Parameter ARNs that Karpenter controller is allowed read access (for retrieving AMI IDs) | `list(string)` | `[]` | no |
| <a name="input_cluster_ip_family"></a> [cluster\_ip\_family](#input\_cluster\_ip\_family) | The IP family used to assign Kubernetes pod and service addresses. Valid values are `ipv4` (default) and `ipv6`. Note: If `ipv6` is specified, the `AmazonEKS_CNI_IPv6_Policy` must exist in the account. This policy is created by the EKS module with `create_cni_ipv6_iam_policy = true` | `string` | `null` | no |
| <a name="input_cluster_name"></a> [cluster\_name](#input\_cluster\_name) | The name of the EKS cluster | `string` | `""` | no |
| <a name="input_create"></a> [create](#input\_create) | Determines whether to create EKS managed node group or not | `bool` | `true` | no |
| <a name="input_create_iam_role"></a> [create\_iam\_role](#input\_create\_iam\_role) | Determines whether an IAM role is created or to use an existing IAM role | `bool` | `true` | no |
| <a name="input_create_instance_profile"></a> [create\_instance\_profile](#input\_create\_instance\_profile) | Whether to create an IAM instance profile | `bool` | `true` | no |
| <a name="input_create_irsa"></a> [create\_irsa](#input\_create\_irsa) | Determines whether an IAM role for service accounts is created | `bool` | `true` | no |
| <a name="input_enable_karpenter_instance_profile_creation"></a> [enable\_karpenter\_instance\_profile\_creation](#input\_enable\_karpenter\_instance\_profile\_creation) | Determines whether Karpenter will be allowed to create the IAM instance profile (v1beta1) or if Terraform will (v1alpha1) | `bool` | `false` | no |
| <a name="input_create"></a> [create](#input\_create) | Controls if resources should be created (affects nearly all resources) | `bool` | `true` | no |
| <a name="input_create_access_entry"></a> [create\_access\_entry](#input\_create\_access\_entry) | Determines whether an access entry is created for the IAM role used by the node IAM role | `bool` | `true` | no |
| <a name="input_create_iam_role"></a> [create\_iam\_role](#input\_create\_iam\_role) | Determines whether an IAM role is created | `bool` | `true` | no |
| <a name="input_create_instance_profile"></a> [create\_instance\_profile](#input\_create\_instance\_profile) | Whether to create an IAM instance profile | `bool` | `false` | no |
| <a name="input_create_node_iam_role"></a> [create\_node\_iam\_role](#input\_create\_node\_iam\_role) | Determines whether an IAM role is created or to use an existing IAM role | `bool` | `true` | no |
| <a name="input_enable_irsa"></a> [enable\_irsa](#input\_enable\_irsa) | Determines whether to enable support IAM role for service account | `bool` | `false` | no |
| <a name="input_enable_spot_termination"></a> [enable\_spot\_termination](#input\_enable\_spot\_termination) | Determines whether to enable native spot termination handling | `bool` | `true` | no |
| <a name="input_iam_role_additional_policies"></a> [iam\_role\_additional\_policies](#input\_iam\_role\_additional\_policies) | Additional policies to be added to the IAM role | `map(string)` | `{}` | no |
| <a name="input_iam_role_arn"></a> [iam\_role\_arn](#input\_iam\_role\_arn) | Existing IAM role ARN for the IAM instance profile. Required if `create_iam_role` is set to `false` | `string` | `null` | no |
| <a name="input_iam_role_attach_cni_policy"></a> [iam\_role\_attach\_cni\_policy](#input\_iam\_role\_attach\_cni\_policy) | Whether to attach the `AmazonEKS_CNI_Policy`/`AmazonEKS_CNI_IPv6_Policy` IAM policy to the IAM IAM role. WARNING: If set `false` the permissions must be assigned to the `aws-node` DaemonSet pods via another method or nodes will not be able to join the cluster | `bool` | `true` | no |
| <a name="input_iam_role_description"></a> [iam\_role\_description](#input\_iam\_role\_description) | Description of the role | `string` | `null` | no |
| <a name="input_iam_policy_description"></a> [iam\_policy\_description](#input\_iam\_policy\_description) | IAM policy description | `string` | `"Karpenter controller IAM policy"` | no |
| <a name="input_iam_policy_name"></a> [iam\_policy\_name](#input\_iam\_policy\_name) | Name of the IAM policy | `string` | `"KarpenterController"` | no |
| <a name="input_iam_policy_path"></a> [iam\_policy\_path](#input\_iam\_policy\_path) | Path of the IAM policy | `string` | `"/"` | no |
| <a name="input_iam_policy_use_name_prefix"></a> [iam\_policy\_use\_name\_prefix](#input\_iam\_policy\_use\_name\_prefix) | Determines whether the name of the IAM policy (`iam_policy_name`) is used as a prefix | `bool` | `true` | no |
| <a name="input_iam_role_description"></a> [iam\_role\_description](#input\_iam\_role\_description) | IAM role description | `string` | `"Karpenter controller IAM role"` | no |
| <a name="input_iam_role_max_session_duration"></a> [iam\_role\_max\_session\_duration](#input\_iam\_role\_max\_session\_duration) | Maximum API session duration in seconds between 3600 and 43200 | `number` | `null` | no |
| <a name="input_iam_role_name"></a> [iam\_role\_name](#input\_iam\_role\_name) | Name to use on IAM role created | `string` | `null` | no |
| <a name="input_iam_role_path"></a> [iam\_role\_path](#input\_iam\_role\_path) | IAM role path | `string` | `"/"` | no |
| <a name="input_iam_role_permissions_boundary"></a> [iam\_role\_permissions\_boundary](#input\_iam\_role\_permissions\_boundary) | ARN of the policy that is used to set the permissions boundary for the IAM role | `string` | `null` | no |
| <a name="input_iam_role_tags"></a> [iam\_role\_tags](#input\_iam\_role\_tags) | A map of additional tags to add to the IAM role created | `map(string)` | `{}` | no |
| <a name="input_iam_role_use_name_prefix"></a> [iam\_role\_use\_name\_prefix](#input\_iam\_role\_use\_name\_prefix) | Determines whether the IAM role name (`iam_role_name`) is used as a prefix | `bool` | `true` | no |
| <a name="input_iam_role_name"></a> [iam\_role\_name](#input\_iam\_role\_name) | Name of the IAM role | `string` | `"KarpenterController"` | no |
| <a name="input_iam_role_path"></a> [iam\_role\_path](#input\_iam\_role\_path) | Path of the IAM role | `string` | `"/"` | no |
| <a name="input_iam_role_permissions_boundary_arn"></a> [iam\_role\_permissions\_boundary\_arn](#input\_iam\_role\_permissions\_boundary\_arn) | Permissions boundary ARN to use for the IAM role | `string` | `null` | no |
| <a name="input_iam_role_policies"></a> [iam\_role\_policies](#input\_iam\_role\_policies) | Policies to attach to the IAM role in `{'static_name' = 'policy_arn'}` format | `map(string)` | `{}` | no |
| <a name="input_iam_role_tags"></a> [iam\_role\_tags](#input\_iam\_role\_tags) | A map of additional tags to add the the IAM role | `map(any)` | `{}` | no |
| <a name="input_iam_role_use_name_prefix"></a> [iam\_role\_use\_name\_prefix](#input\_iam\_role\_use\_name\_prefix) | Determines whether the name of the IAM role (`iam_role_name`) is used as a prefix | `bool` | `true` | no |
| <a name="input_irsa_assume_role_condition_test"></a> [irsa\_assume\_role\_condition\_test](#input\_irsa\_assume\_role\_condition\_test) | Name of the [IAM condition operator](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_condition_operators.html) to evaluate when assuming the role | `string` | `"StringEquals"` | no |
| <a name="input_irsa_description"></a> [irsa\_description](#input\_irsa\_description) | IAM role for service accounts description | `string` | `"Karpenter IAM role for service account"` | no |
| <a name="input_irsa_max_session_duration"></a> [irsa\_max\_session\_duration](#input\_irsa\_max\_session\_duration) | Maximum API session duration in seconds between 3600 and 43200 | `number` | `null` | no |
| <a name="input_irsa_name"></a> [irsa\_name](#input\_irsa\_name) | Name of IAM role for service accounts | `string` | `null` | no |
| <a name="input_irsa_namespace_service_accounts"></a> [irsa\_namespace\_service\_accounts](#input\_irsa\_namespace\_service\_accounts) | List of `namespace:serviceaccount`pairs to use in trust policy for IAM role for service accounts | `list(string)` | <pre>[<br> "karpenter:karpenter"<br>]</pre> | no |
| <a name="input_irsa_oidc_provider_arn"></a> [irsa\_oidc\_provider\_arn](#input\_irsa\_oidc\_provider\_arn) | OIDC provider arn used in trust policy for IAM role for service accounts | `string` | `""` | no |
| <a name="input_irsa_path"></a> [irsa\_path](#input\_irsa\_path) | Path of IAM role for service accounts | `string` | `"/"` | no |
| <a name="input_irsa_permissions_boundary_arn"></a> [irsa\_permissions\_boundary\_arn](#input\_irsa\_permissions\_boundary\_arn) | Permissions boundary ARN to use for IAM role for service accounts | `string` | `null` | no |
| <a name="input_irsa_policy_name"></a> [irsa\_policy\_name](#input\_irsa\_policy\_name) | Name of IAM policy for service accounts | `string` | `null` | no |
| <a name="input_irsa_ssm_parameter_arns"></a> [irsa\_ssm\_parameter\_arns](#input\_irsa\_ssm\_parameter\_arns) | List of SSM Parameter ARNs that contain AMI IDs launched by Karpenter | `list(string)` | <pre>[<br> "arn:aws:ssm:*:*:parameter/aws/service/*"<br>]</pre> | no |
| <a name="input_irsa_subnet_account_id"></a> [irsa\_subnet\_account\_id](#input\_irsa\_subnet\_account\_id) | Account ID of where the subnets Karpenter will utilize resides. Used when subnets are shared from another account | `string` | `""` | no |
| <a name="input_irsa_tag_key"></a> [irsa\_tag\_key](#input\_irsa\_tag\_key) | Tag key (`{key = value}`) applied to resources launched by Karpenter through the Karpenter provisioner | `string` | `"karpenter.sh/discovery"` | no |
| <a name="input_irsa_tag_values"></a> [irsa\_tag\_values](#input\_irsa\_tag\_values) | Tag values (`{key = value}`) applied to resources launched by Karpenter through the Karpenter provisioner. Defaults to cluster name when not set. | `list(string)` | `[]` | no |
| <a name="input_irsa_tags"></a> [irsa\_tags](#input\_irsa\_tags) | A map of additional tags to add the the IAM role for service accounts | `map(any)` | `{}` | no |
| <a name="input_irsa_use_name_prefix"></a> [irsa\_use\_name\_prefix](#input\_irsa\_use\_name\_prefix) | Determines whether the IAM role for service accounts name (`irsa_name`) is used as a prefix | `bool` | `true` | no |
| <a name="input_policies"></a> [policies](#input\_policies) | Policies to attach to the IAM role in `{'static_name' = 'policy_arn'}` format | `map(string)` | `{}` | no |
| <a name="input_node_iam_role_additional_policies"></a> [node\_iam\_role\_additional\_policies](#input\_node\_iam\_role\_additional\_policies) | Additional policies to be added to the IAM role | `map(string)` | `{}` | no |
| <a name="input_node_iam_role_arn"></a> [node\_iam\_role\_arn](#input\_node\_iam\_role\_arn) | Existing IAM role ARN for the IAM instance profile. Required if `create_iam_role` is set to `false` | `string` | `null` | no |
| <a name="input_node_iam_role_attach_cni_policy"></a> [node\_iam\_role\_attach\_cni\_policy](#input\_node\_iam\_role\_attach\_cni\_policy) | Whether to attach the `AmazonEKS_CNI_Policy`/`AmazonEKS_CNI_IPv6_Policy` IAM policy to the IAM IAM role. WARNING: If set `false` the permissions must be assigned to the `aws-node` DaemonSet pods via another method or nodes will not be able to join the cluster | `bool` | `true` | no |
| <a name="input_node_iam_role_description"></a> [node\_iam\_role\_description](#input\_node\_iam\_role\_description) | Description of the role | `string` | `null` | no |
| <a name="input_node_iam_role_max_session_duration"></a> [node\_iam\_role\_max\_session\_duration](#input\_node\_iam\_role\_max\_session\_duration) | Maximum API session duration in seconds between 3600 and 43200 | `number` | `null` | no |
| <a name="input_node_iam_role_name"></a> [node\_iam\_role\_name](#input\_node\_iam\_role\_name) | Name to use on IAM role created | `string` | `null` | no |
| <a name="input_node_iam_role_path"></a> [node\_iam\_role\_path](#input\_node\_iam\_role\_path) | IAM role path | `string` | `"/"` | no |
| <a name="input_node_iam_role_permissions_boundary"></a> [node\_iam\_role\_permissions\_boundary](#input\_node\_iam\_role\_permissions\_boundary) | ARN of the policy that is used to set the permissions boundary for the IAM role | `string` | `null` | no |
| <a name="input_node_iam_role_tags"></a> [node\_iam\_role\_tags](#input\_node\_iam\_role\_tags) | A map of additional tags to add to the IAM role created | `map(string)` | `{}` | no |
| <a name="input_node_iam_role_use_name_prefix"></a> [node\_iam\_role\_use\_name\_prefix](#input\_node\_iam\_role\_use\_name\_prefix) | Determines whether the IAM role name (`iam_role_name`) is used as a prefix | `bool` | `true` | no |
| <a name="input_queue_kms_data_key_reuse_period_seconds"></a> [queue\_kms\_data\_key\_reuse\_period\_seconds](#input\_queue\_kms\_data\_key\_reuse\_period\_seconds) | The length of time, in seconds, for which Amazon SQS can reuse a data key to encrypt or decrypt messages before calling AWS KMS again | `number` | `null` | no |
| <a name="input_queue_kms_master_key_id"></a> [queue\_kms\_master\_key\_id](#input\_queue\_kms\_master\_key\_id) | The ID of an AWS-managed customer master key (CMK) for Amazon SQS or a custom CMK | `string` | `null` | no |
| <a name="input_queue_managed_sse_enabled"></a> [queue\_managed\_sse\_enabled](#input\_queue\_managed\_sse\_enabled) | Boolean to enable server-side encryption (SSE) of message content with SQS-owned encryption keys | `bool` | `true` | no |
@@ -190,17 +174,18 @@ No modules.
| Name | Description |
|------|-------------|
| <a name="output_event_rules"></a> [event\_rules](#output\_event\_rules) | Map of the event rules created and their attributes |
| <a name="output_iam_role_arn"></a> [iam\_role\_arn](#output\_iam\_role\_arn) | The Amazon Resource Name (ARN) specifying the controller IAM role |
| <a name="output_iam_role_name"></a> [iam\_role\_name](#output\_iam\_role\_name) | The name of the controller IAM role |
| <a name="output_iam_role_unique_id"></a> [iam\_role\_unique\_id](#output\_iam\_role\_unique\_id) | Stable and unique string identifying the controller IAM role |
| <a name="output_instance_profile_arn"></a> [instance\_profile\_arn](#output\_instance\_profile\_arn) | ARN assigned by AWS to the instance profile |
| <a name="output_instance_profile_id"></a> [instance\_profile\_id](#output\_instance\_profile\_id) | Instance profile's ID |
| <a name="output_instance_profile_name"></a> [instance\_profile\_name](#output\_instance\_profile\_name) | Name of the instance profile |
| <a name="output_instance_profile_unique"></a> [instance\_profile\_unique](#output\_instance\_profile\_unique) | Stable and unique string identifying the IAM instance profile |
| <a name="output_irsa_arn"></a> [irsa\_arn](#output\_irsa\_arn) | The Amazon Resource Name (ARN) specifying the IAM role for service accounts |
| <a name="output_irsa_name"></a> [irsa\_name](#output\_irsa\_name) | The name of the IAM role for service accounts |
| <a name="output_irsa_unique_id"></a> [irsa\_unique\_id](#output\_irsa\_unique\_id) | Stable and unique string identifying the IAM role for service accounts |
| <a name="output_node_access_entry_arn"></a> [node\_access\_entry\_arn](#output\_node\_access\_entry\_arn) | Amazon Resource Name (ARN) of the node Access Entry |
| <a name="output_node_iam_role_arn"></a> [node\_iam\_role\_arn](#output\_node\_iam\_role\_arn) | The Amazon Resource Name (ARN) specifying the node IAM role |
| <a name="output_node_iam_role_name"></a> [node\_iam\_role\_name](#output\_node\_iam\_role\_name) | The name of the node IAM role |
| <a name="output_node_iam_role_unique_id"></a> [node\_iam\_role\_unique\_id](#output\_node\_iam\_role\_unique\_id) | Stable and unique string identifying the node IAM role |
| <a name="output_queue_arn"></a> [queue\_arn](#output\_queue\_arn) | The ARN of the SQS queue |
| <a name="output_queue_name"></a> [queue\_name](#output\_queue\_name) | The name of the created Amazon SQS queue |
| <a name="output_queue_url"></a> [queue\_url](#output\_queue\_url) | The URL for the created Amazon SQS queue |
| <a name="output_role_arn"></a> [role\_arn](#output\_role\_arn) | The Amazon Resource Name (ARN) specifying the IAM role |
| <a name="output_role_name"></a> [role\_name](#output\_role\_name) | The name of the IAM role |
| <a name="output_role_unique_id"></a> [role\_unique\_id](#output\_role\_unique\_id) | Stable and unique string identifying the IAM role |
<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

560
modules/karpenter/main.tf
View File

@@ -1,207 +1,406 @@
data "aws_region" "current" {}
data "aws_partition" "current" {}
data "aws_caller_identity" "current" {}
locals {
account_id = data.aws_caller_identity.current.account_id
partition = data.aws_partition.current.partition
dns_suffix = data.aws_partition.current.dns_suffix
partition = data.aws_partition.current.partition
region = data.aws_region.current.name
}
################################################################################
# IAM Role for Service Account (IRSA)
# This is used by the Karpenter controller
# Karpenter controller IAM Role
################################################################################
locals {
create_irsa = var.create && var.create_irsa
irsa_name = coalesce(var.irsa_name, "KarpenterIRSA-${var.cluster_name}")
irsa_policy_name = coalesce(var.irsa_policy_name, local.irsa_name)
create_iam_role = var.create && var.create_iam_role
irsa_oidc_provider_url = replace(var.irsa_oidc_provider_arn, "/^(.*provider/)/", "")
}
data "aws_iam_policy_document" "irsa_assume_role" {
count = local.create_irsa ? 1 : 0
data "aws_iam_policy_document" "controller_assume_role" {
count = local.create_iam_role ? 1 : 0
# Pod Identity
statement {
effect = "Allow"
actions = ["sts:AssumeRoleWithWebIdentity"]
actions = [
"sts:AssumeRole",
"sts:TagSession",
]
principals {
type = "Federated"
identifiers = [var.irsa_oidc_provider_arn]
type = "Service"
identifiers = ["pods.eks.amazonaws.com"]
}
}
condition {
test = var.irsa_assume_role_condition_test
variable = "${local.irsa_oidc_provider_url}:sub"
values = [for sa in var.irsa_namespace_service_accounts : "system:serviceaccount:${sa}"]
}
# IAM Roles for Service Accounts (IRSA)
dynamic "statement" {
for_each = var.enable_irsa ? [1] : []
# https://aws.amazon.com/premiumsupport/knowledge-center/eks-troubleshoot-oidc-and-irsa/?nc1=h_ls
condition {
test = var.irsa_assume_role_condition_test
variable = "${local.irsa_oidc_provider_url}:aud"
values = ["sts.amazonaws.com"]
content {
actions = ["sts:AssumeRoleWithWebIdentity"]
principals {
type = "Federated"
identifiers = [var.irsa_oidc_provider_arn]
}
condition {
test = var.irsa_assume_role_condition_test
variable = "${local.irsa_oidc_provider_url}:sub"
values = [for sa in var.irsa_namespace_service_accounts : "system:serviceaccount:${sa}"]
}
# https://aws.amazon.com/premiumsupport/knowledge-center/eks-troubleshoot-oidc-and-irsa/?nc1=h_ls
condition {
test = var.irsa_assume_role_condition_test
variable = "${local.irsa_oidc_provider_url}:aud"
values = ["sts.amazonaws.com"]
}
}
}
}
resource "aws_iam_role" "irsa" {
count = local.create_irsa ? 1 : 0
resource "aws_iam_role" "controller" {
count = local.create_iam_role ? 1 : 0
name = var.irsa_use_name_prefix ? null : local.irsa_name
name_prefix = var.irsa_use_name_prefix ? "${local.irsa_name}-" : null
path = var.irsa_path
description = var.irsa_description
name = var.iam_role_use_name_prefix ? null : var.iam_role_name
name_prefix = var.iam_role_use_name_prefix ? "${var.iam_role_name}-" : null
path = var.iam_role_path
description = var.iam_role_description
assume_role_policy = data.aws_iam_policy_document.irsa_assume_role[0].json
max_session_duration = var.irsa_max_session_duration
permissions_boundary = var.irsa_permissions_boundary_arn
assume_role_policy = data.aws_iam_policy_document.controller_assume_role[0].json
max_session_duration = var.iam_role_max_session_duration
permissions_boundary = var.iam_role_permissions_boundary_arn
force_detach_policies = true
tags = merge(var.tags, var.irsa_tags)
tags = merge(var.tags, var.iam_role_tags)
}
locals {
irsa_tag_values = coalescelist(var.irsa_tag_values, [var.cluster_name])
}
data "aws_iam_policy_document" "irsa" {
count = local.create_irsa ? 1 : 0
data "aws_iam_policy_document" "controller" {
count = local.create_iam_role ? 1 : 0
statement {
actions = [
"ec2:CreateLaunchTemplate",
"ec2:CreateFleet",
"ec2:CreateTags",
"ec2:DescribeLaunchTemplates",
"ec2:DescribeImages",
"ec2:DescribeInstances",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSubnets",
"ec2:DescribeInstanceTypes",
"ec2:DescribeInstanceTypeOfferings",
"ec2:DescribeAvailabilityZones",
"ec2:DescribeSpotPriceHistory",
"pricing:GetProducts",
]
resources = ["*"]
}
statement {
actions = [
"ec2:TerminateInstances",
"ec2:DeleteLaunchTemplate",
]
resources = ["*"]
condition {
test = "StringEquals"
variable = "ec2:ResourceTag/${var.irsa_tag_key}"
values = local.irsa_tag_values
}
}
statement {
actions = ["ec2:RunInstances"]
resources = [
"arn:${local.partition}:ec2:*:${local.account_id}:launch-template/*",
]
condition {
test = "StringEquals"
variable = "ec2:ResourceTag/${var.irsa_tag_key}"
values = local.irsa_tag_values
}
}
statement {
actions = ["ec2:RunInstances"]
sid = "AllowScopedEC2InstanceActions"
resources = [
"arn:${local.partition}:ec2:*::image/*",
"arn:${local.partition}:ec2:*::snapshot/*",
"arn:${local.partition}:ec2:*:${local.account_id}:instance/*",
"arn:${local.partition}:ec2:*:${local.account_id}:spot-instances-request/*",
"arn:${local.partition}:ec2:*:${local.account_id}:security-group/*",
"arn:${local.partition}:ec2:*:${local.account_id}:volume/*",
"arn:${local.partition}:ec2:*:${local.account_id}:network-interface/*",
"arn:${local.partition}:ec2:*:${coalesce(var.irsa_subnet_account_id, local.account_id)}:subnet/*",
"arn:${local.partition}:ec2:*:*:spot-instances-request/*",
"arn:${local.partition}:ec2:*:*:security-group/*",
"arn:${local.partition}:ec2:*:*:subnet/*",
"arn:${local.partition}:ec2:*:*:launch-template/*",
]
actions = [
"ec2:RunInstances",
"ec2:CreateFleet"
]
}
statement {
sid = "AllowScopedEC2InstanceActionsWithTags"
resources = [
"arn:${local.partition}:ec2:*:*:fleet/*",
"arn:${local.partition}:ec2:*:*:instance/*",
"arn:${local.partition}:ec2:*:*:volume/*",
"arn:${local.partition}:ec2:*:*:network-interface/*",
"arn:${local.partition}:ec2:*:*:launch-template/*",
"arn:${local.partition}:ec2:*:*:spot-instances-request/*",
]
actions = [
"ec2:RunInstances",
"ec2:CreateFleet",
"ec2:CreateLaunchTemplate"
]
condition {
test = "StringEquals"
variable = "aws:RequestTag/kubernetes.io/cluster/${var.cluster_name}"
values = ["owned"]
}
condition {
test = "StringLike"
variable = "aws:RequestTag/karpenter.sh/nodepool"
values = ["*"]
}
}
statement {
sid = "AllowScopedResourceCreationTagging"
resources = [
"arn:${local.partition}:ec2:*:*:fleet/*",
"arn:${local.partition}:ec2:*:*:instance/*",
"arn:${local.partition}:ec2:*:*:volume/*",
"arn:${local.partition}:ec2:*:*:network-interface/*",
"arn:${local.partition}:ec2:*:*:launch-template/*",
"arn:${local.partition}:ec2:*:*:spot-instances-request/*",
]
actions = ["ec2:CreateTags"]
condition {
test = "StringEquals"
variable = "aws:RequestTag/kubernetes.io/cluster/${var.cluster_name}"
values = ["owned"]
}
condition {
test = "StringEquals"
variable = "ec2:CreateAction"
values = [
"RunInstances",
"CreateFleet",
"CreateLaunchTemplate",
]
}
condition {
test = "StringLike"
variable = "aws:RequestTag/karpenter.sh/nodepool"
values = ["*"]
}
}
statement {
sid = "AllowScopedResourceTagging"
resources = ["arn:${local.partition}:ec2:*:*:instance/*"]
actions = ["ec2:CreateTags"]
condition {
test = "StringEquals"
variable = "aws:ResourceTag/kubernetes.io/cluster/${var.cluster_name}"
values = ["owned"]
}
condition {
test = "StringLike"
variable = "aws:ResourceTag/karpenter.sh/nodepool"
values = ["*"]
}
condition {
test = "ForAllValues:StringEquals"
variable = "aws:TagKeys"
values = [
"karpenter.sh/nodeclaim",
"Name",
]
}
}
statement {
sid = "AllowScopedDeletion"
resources = [
"arn:${local.partition}:ec2:*:*:instance/*",
"arn:${local.partition}:ec2:*:*:launch-template/*"
]
actions = [
"ec2:TerminateInstances",
"ec2:DeleteLaunchTemplate"
]
condition {
test = "StringEquals"
variable = "aws:ResourceTag/kubernetes.io/cluster/${var.cluster_name}"
values = ["owned"]
}
condition {
test = "StringLike"
variable = "aws:ResourceTag/karpenter.sh/nodepool"
values = ["*"]
}
}
statement {
sid = "AllowRegionalReadActions"
resources = ["*"]
actions = [
"ec2:DescribeAvailabilityZones",
"ec2:DescribeImages",
"ec2:DescribeInstances",
"ec2:DescribeInstanceTypeOfferings",
"ec2:DescribeInstanceTypes",
"ec2:DescribeLaunchTemplates",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSpotPriceHistory",
"ec2:DescribeSubnets"
]
condition {
test = "StringEquals"
variable = "aws:RequestedRegion"
values = [local.region]
}
}
statement {
sid = "AllowSSMReadActions"
resources = coalescelist(var.ami_id_ssm_parameter_arns, ["arn:${local.partition}:ssm:${local.region}::parameter/aws/service/*"])
actions = ["ssm:GetParameter"]
resources = var.irsa_ssm_parameter_arns
}
statement {
actions = ["eks:DescribeCluster"]
resources = ["arn:${local.partition}:eks:*:${local.account_id}:cluster/${var.cluster_name}"]
sid = "AllowPricingReadActions"
resources = ["*"]
actions = ["pricing:GetProducts"]
}
statement {
sid = "AllowInterruptionQueueActions"
resources = [aws_sqs_queue.this[0].arn]
actions = [
"sqs:DeleteMessage",
"sqs:GetQueueAttributes",
"sqs:GetQueueUrl",
"sqs:ReceiveMessage"
]
}
statement {
sid = "AllowPassingInstanceRole"
resources = var.create_node_iam_role ? [aws_iam_role.node[0].arn] : [var.node_iam_role_arn]
actions = ["iam:PassRole"]
resources = [var.create_iam_role ? aws_iam_role.this[0].arn : var.iam_role_arn]
}
dynamic "statement" {
for_each = local.enable_spot_termination ? [1] : []
content {
actions = [
"sqs:DeleteMessage",
"sqs:GetQueueUrl",
"sqs:GetQueueAttributes",
"sqs:ReceiveMessage",
]
resources = [aws_sqs_queue.this[0].arn]
condition {
test = "StringEquals"
variable = "iam:PassedToService"
values = ["ec2.amazonaws.com"]
}
}
# TODO - this will be replaced in v20.0 with the scoped policy provided by Karpenter
# https://github.com/aws/karpenter/blob/main/website/content/en/docs/upgrading/v1beta1-controller-policy.json
dynamic "statement" {
for_each = var.enable_karpenter_instance_profile_creation ? [1] : []
statement {
sid = "AllowScopedInstanceProfileCreationActions"
resources = ["*"]
actions = ["iam:CreateInstanceProfile"]
content {
actions = [
"iam:AddRoleToInstanceProfile",
"iam:CreateInstanceProfile",
"iam:DeleteInstanceProfile",
"iam:GetInstanceProfile",
"iam:RemoveRoleFromInstanceProfile",
"iam:TagInstanceProfile",
]
resources = ["*"]
condition {
test = "StringEquals"
variable = "aws:RequestTag/kubernetes.io/cluster/${var.cluster_name}"
values = ["owned"]
}
condition {
test = "StringEquals"
variable = "aws:RequestTag/topology.kubernetes.io/region"
values = [local.region]
}
condition {
test = "StringLike"
variable = "aws:RequestTag/karpenter.k8s.aws/ec2nodeclass"
values = ["*"]
}
}
statement {
sid = "AllowScopedInstanceProfileTagActions"
resources = ["*"]
actions = ["iam:TagInstanceProfile"]
condition {
test = "StringEquals"
variable = "aws:ResourceTag/kubernetes.io/cluster/${var.cluster_name}"
values = ["owned"]
}
condition {
test = "StringEquals"
variable = "aws:ResourceTag/topology.kubernetes.io/region"
values = [local.region]
}
condition {
test = "StringEquals"
variable = "aws:RequestTag/kubernetes.io/cluster/${var.cluster_name}"
values = ["owned"]
}
condition {
test = "StringEquals"
variable = "aws:ResourceTag/topology.kubernetes.io/region"
values = [local.region]
}
condition {
test = "StringLike"
variable = "aws:ResourceTag/karpenter.k8s.aws/ec2nodeclass"
values = ["*"]
}
condition {
test = "StringLike"
variable = "aws:RequestTag/karpenter.k8s.aws/ec2nodeclass"
values = ["*"]
}
}
statement {
sid = "AllowScopedInstanceProfileActions"
resources = ["*"]
actions = [
"iam:AddRoleToInstanceProfile",
"iam:RemoveRoleFromInstanceProfile",
"iam:DeleteInstanceProfile"
]
condition {
test = "StringEquals"
variable = "aws:ResourceTag/kubernetes.io/cluster/${var.cluster_name}"
values = ["owned"]
}
condition {
test = "StringEquals"
variable = "aws:ResourceTag/topology.kubernetes.io/region"
values = [local.region]
}
condition {
test = "StringLike"
variable = "aws:ResourceTag/karpenter.k8s.aws/ec2nodeclass"
values = ["*"]
}
}
statement {
sid = "AllowInstanceProfileReadActions"
resources = ["*"]
actions = ["iam:GetInstanceProfile"]
}
statement {
sid = "AllowAPIServerEndpointDiscovery"
resources = ["arn:${local.partition}:eks:${local.region}:${local.account_id}:cluster/${var.cluster_name}"]
actions = ["eks:DescribeCluster"]
}
}
resource "aws_iam_policy" "irsa" {
count = local.create_irsa ? 1 : 0
resource "aws_iam_policy" "controller" {
count = local.create_iam_role ? 1 : 0
name_prefix = "${local.irsa_policy_name}-"
path = var.irsa_path
description = var.irsa_description
policy = data.aws_iam_policy_document.irsa[0].json
name = var.iam_policy_use_name_prefix ? null : var.iam_policy_name
name_prefix = var.iam_policy_use_name_prefix ? "${var.iam_policy_name}-" : null
path = var.iam_policy_path
description = var.iam_policy_description
policy = data.aws_iam_policy_document.controller[0].json
tags = var.tags
}
resource "aws_iam_role_policy_attachment" "irsa" {
count = local.create_irsa ? 1 : 0
resource "aws_iam_role_policy_attachment" "controller" {
count = local.create_iam_role ? 1 : 0
role = aws_iam_role.irsa[0].name
policy_arn = aws_iam_policy.irsa[0].arn
role = aws_iam_role.controller[0].name
policy_arn = aws_iam_policy.controller[0].arn
}
resource "aws_iam_role_policy_attachment" "irsa_additional" {
for_each = { for k, v in var.policies : k => v if local.create_irsa }
resource "aws_iam_role_policy_attachment" "controller_additional" {
for_each = { for k, v in var.iam_role_policies : k => v if local.create_iam_role }
role = aws_iam_role.irsa[0].name
role = aws_iam_role.controller[0].name
policy_arn = each.value
}
@@ -266,7 +465,7 @@ locals {
detail-type = ["AWS Health Event"]
}
}
spot_interupt = {
spot_interrupt = {
name = "SpotInterrupt"
description = "Karpenter interrupt - EC2 spot instance interruption warning"
event_pattern = {
@@ -320,15 +519,15 @@ resource "aws_cloudwatch_event_target" "this" {
################################################################################
locals {
create_iam_role = var.create && var.create_iam_role
create_node_iam_role = var.create && var.create_node_iam_role
iam_role_name = coalesce(var.iam_role_name, "Karpenter-${var.cluster_name}")
iam_role_policy_prefix = "arn:${local.partition}:iam::aws:policy"
cni_policy = var.cluster_ip_family == "ipv6" ? "arn:${local.partition}:iam::${local.account_id}:policy/AmazonEKS_CNI_IPv6_Policy" : "${local.iam_role_policy_prefix}/AmazonEKS_CNI_Policy"
node_iam_role_name = coalesce(var.node_iam_role_name, "Karpenter-${var.cluster_name}")
node_iam_role_policy_prefix = "arn:${local.partition}:iam::aws:policy"
cni_policy = var.cluster_ip_family == "ipv6" ? "arn:${local.partition}:iam::${local.account_id}:policy/AmazonEKS_CNI_IPv6_Policy" : "${local.node_iam_role_policy_prefix}/AmazonEKS_CNI_Policy"
}
data "aws_iam_policy_document" "assume_role" {
count = local.create_iam_role ? 1 : 0
data "aws_iam_policy_document" "node_assume_role" {
count = local.create_node_iam_role ? 1 : 0
statement {
sid = "EKSNodeAssumeRole"
@@ -336,62 +535,83 @@ data "aws_iam_policy_document" "assume_role" {
principals {
type = "Service"
identifiers = ["ec2.${local.dns_suffix}"]
identifiers = ["ec2.amazonaws.com"]
}
}
}
resource "aws_iam_role" "this" {
count = local.create_iam_role ? 1 : 0
resource "aws_iam_role" "node" {
count = local.create_node_iam_role ? 1 : 0
name = var.iam_role_use_name_prefix ? null : local.iam_role_name
name_prefix = var.iam_role_use_name_prefix ? "${local.iam_role_name}-" : null
path = var.iam_role_path
description = var.iam_role_description
name = var.node_iam_role_use_name_prefix ? null : local.node_iam_role_name
name_prefix = var.node_iam_role_use_name_prefix ? "${local.node_iam_role_name}-" : null
path = var.node_iam_role_path
description = var.node_iam_role_description
assume_role_policy = data.aws_iam_policy_document.assume_role[0].json
max_session_duration = var.iam_role_max_session_duration
permissions_boundary = var.iam_role_permissions_boundary
assume_role_policy = data.aws_iam_policy_document.node_assume_role[0].json
max_session_duration = var.node_iam_role_max_session_duration
permissions_boundary = var.node_iam_role_permissions_boundary
force_detach_policies = true
tags = merge(var.tags, var.iam_role_tags)
tags = merge(var.tags, var.node_iam_role_tags)
}
# Policies attached ref https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_node_group
resource "aws_iam_role_policy_attachment" "this" {
resource "aws_iam_role_policy_attachment" "node" {
for_each = { for k, v in toset(compact([
"${local.iam_role_policy_prefix}/AmazonEKSWorkerNodePolicy",
"${local.iam_role_policy_prefix}/AmazonEC2ContainerRegistryReadOnly",
var.iam_role_attach_cni_policy ? local.cni_policy : "",
])) : k => v if local.create_iam_role }
"${local.node_iam_role_policy_prefix}/AmazonEKSWorkerNodePolicy",
"${local.node_iam_role_policy_prefix}/AmazonEC2ContainerRegistryReadOnly",
var.node_iam_role_attach_cni_policy ? local.cni_policy : "",
])) : k => v if local.create_node_iam_role }
policy_arn = each.value
role = aws_iam_role.this[0].name
role = aws_iam_role.node[0].name
}
resource "aws_iam_role_policy_attachment" "additional" {
for_each = { for k, v in var.iam_role_additional_policies : k => v if local.create_iam_role }
resource "aws_iam_role_policy_attachment" "node_additional" {
for_each = { for k, v in var.node_iam_role_additional_policies : k => v if local.create_node_iam_role }
policy_arn = each.value
role = aws_iam_role.this[0].name
role = aws_iam_role.node[0].name
}
################################################################################
# Access Entry
################################################################################
resource "aws_eks_access_entry" "node" {
count = var.create && var.create_access_entry ? 1 : 0
cluster_name = var.cluster_name
principal_arn = var.create_node_iam_role ? aws_iam_role.node[0].arn : var.node_iam_role_arn
type = var.access_entry_type
tags = var.tags
depends_on = [
# If we try to add this too quickly, it fails. So .... we wait
aws_sqs_queue_policy.this,
]
}
################################################################################
# Node IAM Instance Profile
# This is used by the nodes launched by Karpenter
# Starting with Karpenter 0.32 this is no longer required as Karpenter will
# create the Instance Profile
################################################################################
locals {
external_role_name = try(replace(var.iam_role_arn, "/^(.*role/)/", ""), null)
external_role_name = try(replace(var.node_iam_role_arn, "/^(.*role/)/", ""), null)
}
resource "aws_iam_instance_profile" "this" {
count = var.create && var.create_instance_profile ? 1 : 0
name = var.iam_role_use_name_prefix ? null : local.iam_role_name
name_prefix = var.iam_role_use_name_prefix ? "${local.iam_role_name}-" : null
path = var.iam_role_path
role = var.create_iam_role ? aws_iam_role.this[0].name : local.external_role_name
name = var.node_iam_role_use_name_prefix ? null : local.node_iam_role_name
name_prefix = var.node_iam_role_use_name_prefix ? "${local.node_iam_role_name}-" : null
path = var.node_iam_role_path
role = var.create_node_iam_role ? aws_iam_role.node[0].name : local.external_role_name
tags = merge(var.tags, var.iam_role_tags)
tags = merge(var.tags, var.node_iam_role_tags)
}

56
modules/karpenter/migrations.tf Normal file
View File

@@ -0,0 +1,56 @@
################################################################################
# Migrations: v19.21 -> v20.0
################################################################################
# Node IAM role
moved {
from = aws_iam_role.this
to = aws_iam_role.node
}
moved {
from = aws_iam_policy.this
to = aws_iam_policy.node
}
moved {
from = aws_iam_role_policy_attachment.this
to = aws_iam_role_policy_attachment.node
}
moved {
from = aws_iam_role_policy_attachment.additional
to = aws_iam_role_policy_attachment.node_additional
}
# Controller IAM role
moved {
from = aws_iam_role.irsa
to = aws_iam_role.controller
}
moved {
from = aws_iam_policy.irsa
to = aws_iam_policy.controller
}
moved {
from = aws_iam_role_policy_attachment.irsa
to = aws_iam_role_policy_attachment.controller
}
moved {
from = aws_iam_role_policy_attachment.irsa_additional
to = aws_iam_role_policy_attachment.controller_additional
}
# Spelling correction
moved {
from = aws_cloudwatch_event_target.this["spot_interupt"]
to = aws_cloudwatch_event_target.this["spot_interrupt"]
}
moved {
from = aws_cloudwatch_event_rule.this["spot_interupt"]
to = aws_cloudwatch_event_rule.this["spot_interrupt"]
}

47
modules/karpenter/outputs.tf
View File

@@ -1,20 +1,20 @@
################################################################################
# IAM Role for Service Account (IRSA)
# Karpenter controller IAM Role
################################################################################
output "irsa_name" {
description = "The name of the IAM role for service accounts"
value = try(aws_iam_role.irsa[0].name, null)
output "iam_role_name" {
description = "The name of the controller IAM role"
value = try(aws_iam_role.controller[0].name, null)
}
output "irsa_arn" {
description = "The Amazon Resource Name (ARN) specifying the IAM role for service accounts"
value = try(aws_iam_role.irsa[0].arn, null)
output "iam_role_arn" {
description = "The Amazon Resource Name (ARN) specifying the controller IAM role"
value = try(aws_iam_role.controller[0].arn, null)
}
output "irsa_unique_id" {
description = "Stable and unique string identifying the IAM role for service accounts"
value = try(aws_iam_role.irsa[0].unique_id, null)
output "iam_role_unique_id" {
description = "Stable and unique string identifying the controller IAM role"
value = try(aws_iam_role.controller[0].unique_id, null)
}
################################################################################
@@ -49,19 +49,28 @@ output "event_rules" {
# Node IAM Role
################################################################################
output "role_name" {
description = "The name of the IAM role"
value = try(aws_iam_role.this[0].name, null)
output "node_iam_role_name" {
description = "The name of the node IAM role"
value = try(aws_iam_role.node[0].name, null)
}
output "role_arn" {
description = "The Amazon Resource Name (ARN) specifying the IAM role"
value = try(aws_iam_role.this[0].arn, var.iam_role_arn)
output "node_iam_role_arn" {
description = "The Amazon Resource Name (ARN) specifying the node IAM role"
value = try(aws_iam_role.node[0].arn, var.node_iam_role_arn)
}
output "role_unique_id" {
description = "Stable and unique string identifying the IAM role"
value = try(aws_iam_role.this[0].unique_id, null)
output "node_iam_role_unique_id" {
description = "Stable and unique string identifying the node IAM role"
value = try(aws_iam_role.node[0].unique_id, null)
}
################################################################################
# Access Entry
################################################################################
output "node_access_entry_arn" {
description = "Amazon Resource Name (ARN) of the node Access Entry"
value = try(aws_eks_access_entry.node[0].access_entry_arn, null)
}
################################################################################

147
modules/karpenter/variables.tf
View File

@@ -1,5 +1,5 @@
variable "create" {
description = "Determines whether to create EKS managed node group or not"
description = "Controls if resources should be created (affects nearly all resources)"
type = bool
default = true
}
@@ -17,92 +17,101 @@ variable "cluster_name" {
}
################################################################################
# IAM Role for Service Account (IRSA)
# Karpenter controller IAM Role
################################################################################
variable "create_irsa" {
description = "Determines whether an IAM role for service accounts is created"
variable "create_iam_role" {
description = "Determines whether an IAM role is created"
type = bool
default = true
}
variable "irsa_name" {
description = "Name of IAM role for service accounts"
variable "iam_role_name" {
description = "Name of the IAM role"
type = string
default = null
default = "KarpenterController"
}
variable "irsa_policy_name" {
description = "Name of IAM policy for service accounts"
type = string
default = null
}
variable "irsa_use_name_prefix" {
description = "Determines whether the IAM role for service accounts name (`irsa_name`) is used as a prefix"
variable "iam_role_use_name_prefix" {
description = "Determines whether the name of the IAM role (`iam_role_name`) is used as a prefix"
type = bool
default = true
}
variable "irsa_path" {
description = "Path of IAM role for service accounts"
variable "iam_role_path" {
description = "Path of the IAM role"
type = string
default = "/"
}
variable "irsa_description" {
description = "IAM role for service accounts description"
variable "iam_role_description" {
description = "IAM role description"
type = string
default = "Karpenter IAM role for service account"
default = "Karpenter controller IAM role"
}
variable "irsa_max_session_duration" {
variable "iam_role_max_session_duration" {
description = "Maximum API session duration in seconds between 3600 and 43200"
type = number
default = null
}
variable "irsa_permissions_boundary_arn" {
description = "Permissions boundary ARN to use for IAM role for service accounts"
variable "iam_role_permissions_boundary_arn" {
description = "Permissions boundary ARN to use for the IAM role"
type = string
default = null
}
variable "irsa_tags" {
description = "A map of additional tags to add the the IAM role for service accounts"
variable "iam_role_tags" {
description = "A map of additional tags to add the the IAM role"
type = map(any)
default = {}
}
variable "policies" {
variable "iam_policy_name" {
description = "Name of the IAM policy"
type = string
default = "KarpenterController"
}
variable "iam_policy_use_name_prefix" {
description = "Determines whether the name of the IAM policy (`iam_policy_name`) is used as a prefix"
type = bool
default = true
}
variable "iam_policy_path" {
description = "Path of the IAM policy"
type = string
default = "/"
}
variable "iam_policy_description" {
description = "IAM policy description"
type = string
default = "Karpenter controller IAM policy"
}
variable "iam_role_policies" {
description = "Policies to attach to the IAM role in `{'static_name' = 'policy_arn'}` format"
type = map(string)
default = {}
}
variable "irsa_tag_key" {
description = "Tag key (`{key = value}`) applied to resources launched by Karpenter through the Karpenter provisioner"
type = string
default = "karpenter.sh/discovery"
}
variable "irsa_tag_values" {
description = "Tag values (`{key = value}`) applied to resources launched by Karpenter through the Karpenter provisioner. Defaults to cluster name when not set."
variable "ami_id_ssm_parameter_arns" {
description = "List of SSM Parameter ARNs that Karpenter controller is allowed read access (for retrieving AMI IDs)"
type = list(string)
default = []
}
variable "irsa_ssm_parameter_arns" {
description = "List of SSM Parameter ARNs that contain AMI IDs launched by Karpenter"
type = list(string)
# https://github.com/aws/karpenter/blob/ed9473a9863ca949b61b9846c8b9f33f35b86dbd/pkg/cloudprovider/aws/ami.go#L105-L123
default = ["arn:aws:ssm:*:*:parameter/aws/service/*"]
}
################################################################################
# IAM Role for Service Account (IRSA)
################################################################################
variable "irsa_subnet_account_id" {
description = "Account ID of where the subnets Karpenter will utilize resides. Used when subnets are shared from another account"
type = string
default = ""
variable "enable_irsa" {
description = "Determines whether to enable support IAM role for service account"
type = bool
default = false
}
variable "irsa_oidc_provider_arn" {
@@ -123,12 +132,6 @@ variable "irsa_assume_role_condition_test" {
default = "StringEquals"
}
variable "enable_karpenter_instance_profile_creation" {
description = "Determines whether Karpenter will be allowed to create the IAM instance profile (v1beta1) or if Terraform will (v1alpha1)"
type = bool
default = false
}
################################################################################
# Node Termination Queue
################################################################################
@@ -164,81 +167,97 @@ variable "queue_kms_data_key_reuse_period_seconds" {
}
################################################################################
# Node IAM Role & Instance Profile
# Node IAM Role
################################################################################
variable "create_iam_role" {
variable "create_node_iam_role" {
description = "Determines whether an IAM role is created or to use an existing IAM role"
type = bool
default = true
}
variable "cluster_ip_family" {
description = "The IP family used to assign Kubernetes pod and service addresses. Valid values are `ipv4` (default) and `ipv6`"
description = "The IP family used to assign Kubernetes pod and service addresses. Valid values are `ipv4` (default) and `ipv6`. Note: If `ipv6` is specified, the `AmazonEKS_CNI_IPv6_Policy` must exist in the account. This policy is created by the EKS module with `create_cni_ipv6_iam_policy = true`"
type = string
default = null
}
variable "iam_role_arn" {
variable "node_iam_role_arn" {
description = "Existing IAM role ARN for the IAM instance profile. Required if `create_iam_role` is set to `false`"
type = string
default = null
}
variable "iam_role_name" {
variable "node_iam_role_name" {
description = "Name to use on IAM role created"
type = string
default = null
}
variable "iam_role_use_name_prefix" {
variable "node_iam_role_use_name_prefix" {
description = "Determines whether the IAM role name (`iam_role_name`) is used as a prefix"
type = bool
default = true
}
variable "iam_role_path" {
variable "node_iam_role_path" {
description = "IAM role path"
type = string
default = "/"
}
variable "iam_role_description" {
variable "node_iam_role_description" {
description = "Description of the role"
type = string
default = null
}
variable "iam_role_max_session_duration" {
variable "node_iam_role_max_session_duration" {
description = "Maximum API session duration in seconds between 3600 and 43200"
type = number
default = null
}
variable "iam_role_permissions_boundary" {
variable "node_iam_role_permissions_boundary" {
description = "ARN of the policy that is used to set the permissions boundary for the IAM role"
type = string
default = null
}
variable "iam_role_attach_cni_policy" {
variable "node_iam_role_attach_cni_policy" {
description = "Whether to attach the `AmazonEKS_CNI_Policy`/`AmazonEKS_CNI_IPv6_Policy` IAM policy to the IAM IAM role. WARNING: If set `false` the permissions must be assigned to the `aws-node` DaemonSet pods via another method or nodes will not be able to join the cluster"
type = bool
default = true
}
variable "iam_role_additional_policies" {
variable "node_iam_role_additional_policies" {
description = "Additional policies to be added to the IAM role"
type = map(string)
default = {}
}
variable "iam_role_tags" {
variable "node_iam_role_tags" {
description = "A map of additional tags to add to the IAM role created"
type = map(string)
default = {}
}
################################################################################
# Access Entry
################################################################################
variable "create_access_entry" {
description = "Determines whether an access entry is created for the IAM role used by the node IAM role"
type = bool
default = true
}
variable "access_entry_type" {
description = "Type of the access entry. `EC2_LINUX`, `FARGATE_LINUX`, or `EC2_WINDOWS`; defaults to `EC2_LINUX`"
type = string
default = "EC2_LINUX"
}
################################################################################
# Node IAM Instance Profile
################################################################################
@@ -246,7 +265,7 @@ variable "iam_role_tags" {
variable "create_instance_profile" {
description = "Whether to create an IAM instance profile"
type = bool
default = true
default = false
}
################################################################################

4
modules/karpenter/versions.tf
View File

@@ -1,10 +1,10 @@
terraform {
required_version = ">= 1.0"
required_version = ">= 1.3"
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 4.57"
version = ">= 5.34"
}
}
}