github-mirrors/terraform-aws-eks
1
0
Fork 0
mirror of https://github.com/ysoftdevs/terraform-aws-eks.git synced 2026-03-14 14:21:13 +01:00
Code Issues Packages Projects Releases Wiki Activity

Allows worker groups to be created using AWS Launch Templates (#222)

Browse Source
This commit is contained in:
skang0601
2019-01-10 09:19:19 -06:00
committed by Max Williams
parent 646831423a
commit f3ab7c67da
10 changed files with 218 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
CHANGELOG.md
View File

@@ -33,6 +33,7 @@ project adheres to [Semantic Versioning](http://semver.org/).
### Added
- Support for using AWS Launch Templates to define autoscaling groups (by @skang0601)
- `suspended_processes` to `worker_groups` input (by @bkmeneguello)
- `target_group_arns` to `worker_groups` input (by @zihaoyu)
- `force_detach_policies` to `aws_iam_role` `cluster` and `workers` (by @marky-mark)
@@ -41,6 +42,7 @@ project adheres to [Semantic Versioning](http://semver.org/).
### Changed
- new variables worker_groups_launch_template and worker_group_count_launch_template (by @skang0601)
- Remove aws_iam_service_linked_role (by @max-rocket-internet)
- Adjust the order and correct/update the ec2 instance type info. (@chenrui333)
- Removed providers from `main.tf`. (by @max-rocket-internet)

6
README.md
View File

@@ -136,10 +136,13 @@ MIT Licensed. See [LICENSE](https://github.com/terraform-aws-modules/terraform-a
| worker\_additional\_security\_group\_ids | A list of additional security group ids to attach to worker instances | list | `[]` | no |
| worker\_create\_security\_group | Whether to create a security group for the workers or attach the workers to `worker_security_group_id`. | string | `true` | no |
| worker\_group\_count | The number of maps contained within the worker_groups list. | string | `1` | no |
| worker\_groups | A list of maps defining worker group configurations. See workers_group_defaults for valid keys. | list | `[ { "name": "default" } ]` | no |
| worker\_group\_launch\_template\_count | The number of maps contained within the worker_groups_launch_template list. | string | `0` | no |
| worker\_groups | A list of maps defining worker group configurations to be defined using AWS Launch Configurations. See workers_group_defaults for valid keys. | list | `[ { "name": "default" } ]` | no |
| worker\_groups\_launch\_template | A list of maps defining worker group configurations to be defined using AWS Launch Templates. See workers_group_defaults for valid keys. | list | `[ { "name": "default" } ]` | no |
| worker\_security\_group\_id | If provided, all workers will be attached to this security group. If not given, a security group will be created with necessary ingres/egress to work with the EKS cluster. | string | `` | no |
| worker\_sg\_ingress\_from\_port | Minimum port number from which pods will accept communication. Must be changed to a lower value if some pods in your cluster will expose a port lower than 1025 (e.g. 22, 80, or 443). | string | `1025` | no |
| workers\_group\_defaults | Override default values for target groups. See workers_group_defaults_defaults in locals.tf for valid keys. | map | `{}` | no |
| workers\_group\_launch\_template\_defaults | Override default values for target groups. See workers_group_defaults_defaults in locals.tf for valid keys. | map | `{}` | no |
| write\_kubeconfig | Whether to write a Kubectl config file containing the cluster configuration. Saved to `config_output_path`. | string | `true` | no |
## Outputs
@@ -158,4 +161,5 @@ MIT Licensed. See [LICENSE](https://github.com/terraform-aws-modules/terraform-a
| worker\_security\_group\_id | Security group ID attached to the EKS workers. |
| workers\_asg\_arns | IDs of the autoscaling groups containing workers. |
| workers\_asg\_names | Names of the autoscaling groups containing workers. |
<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

11
aws_auth.tf
View File

@@ -21,6 +21,15 @@ resource "null_resource" "update_config_map_aws_auth" {
data "aws_caller_identity" "current" {}
data "template_file" "launch_template_worker_role_arns" {
count = "${var.worker_group_launch_template_count}"
template = "${file("${path.module}/templates/worker-role.tpl")}"
vars {
worker_role_arn = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:role/${element(aws_iam_instance_profile.workers_launch_template.*.role, count.index)}"
}
}
data "template_file" "worker_role_arns" {
count = "${var.worker_group_count}"
template = "${file("${path.module}/templates/worker-role.tpl")}"
@@ -34,7 +43,7 @@ data "template_file" "config_map_aws_auth" {
template = "${file("${path.module}/templates/config-map-aws-auth.yaml.tpl")}"
vars {
worker_role_arn = "${join("", distinct(data.template_file.worker_role_arns.*.rendered))}"
worker_role_arn = "${join("", distinct(concat(data.template_file.launch_template_worker_role_arns.*.rendered, data.template_file.worker_role_arns.*.rendered)))}"
map_users = "${join("", data.template_file.map_users.*.rendered)}"
map_roles = "${join("", data.template_file.map_roles.*.rendered)}"
map_accounts = "${join("", data.template_file.map_accounts.*.rendered)}"

14
data.tf
View File

@@ -82,3 +82,17 @@ data "template_file" "userdata" {
kubelet_extra_args = "${lookup(var.worker_groups[count.index], "kubelet_extra_args", local.workers_group_defaults["kubelet_extra_args"])}"
}
}
data "template_file" "launch_template_userdata" {
template = "${file("${path.module}/templates/userdata.sh.tpl")}"
count = "${var.worker_group_launch_template_count}"
vars {
cluster_name = "${aws_eks_cluster.this.name}"
endpoint = "${aws_eks_cluster.this.endpoint}"
cluster_auth_base64 = "${aws_eks_cluster.this.certificate_authority.0.data}"
pre_userdata = "${lookup(var.worker_groups_launch_template[count.index], "pre_userdata", local.workers_group_defaults["pre_userdata"])}"
additional_userdata = "${lookup(var.worker_groups_launch_template[count.index], "additional_userdata", local.workers_group_defaults["additional_userdata"])}"
kubelet_extra_args = "${lookup(var.worker_groups_launch_template[count.index], "kubelet_extra_args", local.workers_group_defaults["kubelet_extra_args"])}"
}
}

29
examples/eks_test_fixture/main.tf
View File

@@ -3,7 +3,7 @@ terraform {
}
provider "aws" {
version = ">= 1.24.0"
version = ">= 1.47.0"
region = "${var.region}"
}
@@ -41,15 +41,24 @@ locals {
worker_groups = [
{
instance_type = "t2.small"
additional_userdata = "echo foo bar"
subnets = "${join(",", module.vpc.private_subnets)}"
# This will launch an autoscaling group with only On-Demand instances
instance_type = "t2.small"
additional_userdata = "echo foo bar"
subnets = "${join(",", module.vpc.private_subnets)}"
asg_desired_capacity = "2"
},
]
worker_groups_launch_template = [
{
instance_type = "t2.small"
additional_userdata = "echo foo bar"
subnets = "${join(",", module.vpc.private_subnets)}"
additional_security_group_ids = "${aws_security_group.worker_group_mgmt_one.id},${aws_security_group.worker_group_mgmt_two.id}"
# This will launch an autoscaling group with only Spot Fleet instances
instance_type = "t2.small"
additional_userdata = "echo foo bar"
subnets = "${join(",", module.vpc.private_subnets)}"
additional_security_group_ids = "${aws_security_group.worker_group_mgmt_one.id},${aws_security_group.worker_group_mgmt_two.id}"
override_instance_type = "t3.small"
asg_desired_capacity = "2"
spot_instance_pools = 10
on_demand_percentage_above_base_capacity = "0"
},
]
tags = {
@@ -133,7 +142,9 @@ module "eks" {
tags = "${local.tags}"
vpc_id = "${module.vpc.vpc_id}"
worker_groups = "${local.worker_groups}"
worker_group_count = "2"
worker_groups_launch_template = "${local.worker_groups_launch_template}"
worker_group_count = "1"
worker_group_launch_template_count = "1"
worker_additional_security_group_ids = ["${aws_security_group.all_worker_mgmt.id}"]
map_roles = "${var.map_roles}"
map_roles_count = "${var.map_roles_count}"

38
local.tf
View File

@@ -39,6 +39,44 @@ locals {
workers_group_defaults = "${merge(local.workers_group_defaults_defaults, var.workers_group_defaults)}"
workers_group_launch_template_defaults_defaults = {
name = "count.index" # Name of the worker group. Literal count.index will never be used but if name is not set, the count.index interpolation will be used.
ami_id = "${data.aws_ami.eks_worker.id}" # AMI ID for the eks workers. If none is provided, Terraform will search for the latest version of their EKS optimized worker AMI.
root_block_device_id = "${data.aws_ami.eks_worker.root_device_name}" # Root device name for workers. If non is provided, will assume default AMI was used.
asg_desired_capacity = "1" # Desired worker capacity in the autoscaling group.
asg_max_size = "3" # Maximum worker capacity in the autoscaling group.
asg_min_size = "1" # Minimum worker capacity in the autoscaling group.
instance_type = "m4.large" # Size of the workers instances.
override_instance_type = "t3.large" # Need to specify at least one additional instance type for mixed instances policy. The instance_type holds higher priority for on demand instances.
on_demand_allocation_strategy = "prioritized" # Strategy to use when launching on-demand instances. Valid values: prioritized.
on_demand_base_capacity = "0" # Absolute minimum amount of desired capacity that must be fulfilled by on-demand instances
on_demand_percentage_above_base_capacity = "100" # Percentage split between on-demand and Spot instances above the base on-demand capacity
spot_allocation_strategy = "lowest-price" # The only valid value is lowest-price, which is also the default value. The Auto Scaling group selects the cheapest Spot pools and evenly allocates your Spot capacity across the number of Spot pools that you specify.
spot_instance_pools = 10 # "Number of Spot pools per availability zone to allocate capacity. EC2 Auto Scaling selects the cheapest Spot pools and evenly allocates Spot capacity across the number of Spot pools that you specify."
spot_max_price = "" # Maximum price per unit hour that the user is willing to pay for the Spot instances. Default is the on-demand price
spot_price = "" # Cost of spot instance.
placement_tenancy = "default" # The tenancy of the instance. Valid values are "default" or "dedicated".
root_volume_size = "100" # root volume size of workers instances.
root_volume_type = "gp2" # root volume type of workers instances, can be 'standard', 'gp2', or 'io1'
root_iops = "0" # The amount of provisioned IOPS. This must be set with a volume_type of "io1".
key_name = "" # The key name that should be used for the instances in the autoscaling group
pre_userdata = "" # userdata to pre-append to the default userdata.
additional_userdata = "" # userdata to append to the default userdata.
ebs_optimized = true # sets whether to use ebs optimization on supported types.
enable_monitoring = true # Enables/disables detailed monitoring.
public_ip = false # Associate a public ip address with a worker
kubelet_extra_args = "" # This string is passed directly to kubelet if set. Useful for adding labels or taints.
subnets = "${join(",", var.subnets)}" # A comma delimited string of subnets to place the worker nodes in. i.e. subnet-123,subnet-456,subnet-789
autoscaling_enabled = false # Sets whether policy and matching tags will be added to allow autoscaling.
additional_security_group_ids = "" # A comma delimited list of additional security group ids to include in worker launch config
protect_from_scale_in = false # Prevent AWS from scaling in, so that cluster-autoscaler is solely responsible.
iam_role_id = "${local.default_iam_role_id}" # Use the specified IAM role if set.
suspended_processes = "" # A comma delimited string of processes to to suspend. i.e. AZRebalance,HealthCheck,ReplaceUnhealthy
target_group_arns = "" # A comma delimited list of ALB target group ARNs to be associated to the ASG
}
workers_group_launch_template_defaults = "${merge(local.workers_group_launch_template_defaults_defaults, var.workers_group_launch_template_defaults)}"
ebs_optimized = {
"c1.medium" = false
"c1.xlarge" = true

4
outputs.tf
View File

@@ -41,12 +41,12 @@ output "kubeconfig" {
output "workers_asg_arns" {
description = "IDs of the autoscaling groups containing workers."
value = "${aws_autoscaling_group.workers.*.arn}"
value = "${concat(aws_autoscaling_group.workers.*.arn, aws_autoscaling_group.workers_launch_template.*.arn)}"
}
output "workers_asg_names" {
description = "Names of the autoscaling groups containing workers."
value = "${aws_autoscaling_group.workers.*.id}"
value = "${concat(aws_autoscaling_group.workers.*.id, aws_autoscaling_group.workers_launch_template.*.id)}"
}
output "worker_security_group_id" {

23
variables.tf
View File

@@ -79,7 +79,7 @@ variable "vpc_id" {
}
variable "worker_groups" {
description = "A list of maps defining worker group configurations. See workers_group_defaults for valid keys."
description = "A list of maps defining worker group configurations to be defined using AWS Launch Configurations. See workers_group_defaults for valid keys."
type = "list"
default = [{
@@ -99,6 +99,27 @@ variable "workers_group_defaults" {
default = {}
}
variable "worker_groups_launch_template" {
description = "A list of maps defining worker group configurations to be defined using AWS Launch Templates. See workers_group_defaults for valid keys."
type = "list"
default = [{
"name" = "default"
}]
}
variable "worker_group_launch_template_count" {
description = "The number of maps contained within the worker_groups_launch_template list."
type = "string"
default = "0"
}
variable "workers_group_launch_template_defaults" {
description = "Override default values for target groups. See workers_group_defaults_defaults in locals.tf for valid keys."
type = "map"
default = {}
}
variable "worker_security_group_id" {
description = "If provided, all workers will be attached to this security group. If not given, a security group will be created with necessary ingres/egress to work with the EKS cluster."
default = ""

2
workers.tf
View File

@@ -1,3 +1,5 @@
# Worker Groups using Launch Configurations
resource "aws_autoscaling_group" "workers" {
name_prefix = "${aws_eks_cluster.this.name}-${lookup(var.worker_groups[count.index], "name", count.index)}"
desired_capacity = "${lookup(var.worker_groups[count.index], "asg_desired_capacity", local.workers_group_defaults["asg_desired_capacity"])}"

103
workers_launch_template.tf Normal file
View File

@@ -0,0 +1,103 @@
# Worker Groups using Launch Templates
resource "aws_autoscaling_group" "workers_launch_template" {
name_prefix = "${aws_eks_cluster.this.name}-${lookup(var.worker_groups_launch_template[count.index], "name", count.index)}"
desired_capacity = "${lookup(var.worker_groups_launch_template[count.index], "asg_desired_capacity", local.workers_group_launch_template_defaults["asg_desired_capacity"])}"
max_size = "${lookup(var.worker_groups_launch_template[count.index], "asg_max_size", local.workers_group_launch_template_defaults["asg_max_size"])}"
min_size = "${lookup(var.worker_groups_launch_template[count.index], "asg_min_size", local.workers_group_launch_template_defaults["asg_min_size"])}"
target_group_arns = ["${compact(split(",", coalesce(lookup(var.worker_groups_launch_template[count.index], "target_group_arns", ""), local.workers_group_launch_template_defaults["target_group_arns"])))}"]
mixed_instances_policy {
instances_distribution {
on_demand_allocation_strategy = "${lookup(var.worker_groups_launch_template[count.index], "on_demand_allocation_strategy", local.workers_group_launch_template_defaults["on_demand_allocation_strategy"])}"
on_demand_base_capacity = "${lookup(var.worker_groups_launch_template[count.index], "on_demand_base_capacity", local.workers_group_launch_template_defaults["on_demand_base_capacity"])}"
on_demand_percentage_above_base_capacity = "${lookup(var.worker_groups_launch_template[count.index], "on_demand_percentage_above_base_capacity", local.workers_group_launch_template_defaults["on_demand_percentage_above_base_capacity"])}"
spot_allocation_strategy = "${lookup(var.worker_groups_launch_template[count.index], "spot_allocation_strategy", local.workers_group_launch_template_defaults["spot_allocation_strategy"])}"
spot_instance_pools = "${lookup(var.worker_groups_launch_template[count.index], "spot_instance_pools", local.workers_group_launch_template_defaults["spot_instance_pools"])}"
spot_max_price = "${lookup(var.worker_groups_launch_template[count.index], "spot_max_price", local.workers_group_launch_template_defaults["spot_max_price"])}"
}
launch_template {
launch_template_specification {
launch_template_id = "${element(aws_launch_template.workers_launch_template.*.id, count.index)}"
version = "$Latest"
}
override {
instance_type = "${lookup(var.worker_groups_launch_template[count.index], "instance_type", local.workers_group_launch_template_defaults["instance_type"])}"
}
override {
instance_type = "${lookup(var.worker_groups_launch_template[count.index], "override_instance_type", local.workers_group_launch_template_defaults["override_instance_type"])}"
}
}
}
vpc_zone_identifier = ["${split(",", coalesce(lookup(var.worker_groups_launch_template[count.index], "subnets", ""), local.workers_group_launch_template_defaults["subnets"]))}"]
protect_from_scale_in = "${lookup(var.worker_groups_launch_template[count.index], "protect_from_scale_in", local.workers_group_launch_template_defaults["protect_from_scale_in"])}"
suspended_processes = ["${compact(split(",", coalesce(lookup(var.worker_groups_launch_template[count.index], "suspended_processes", ""), local.workers_group_launch_template_defaults["suspended_processes"])))}"]
count = "${var.worker_group_launch_template_count}"
tags = ["${concat(
list(
map("key", "Name", "value", "${aws_eks_cluster.this.name}-${lookup(var.worker_groups_launch_template[count.index], "name", count.index)}-eks_asg", "propagate_at_launch", true),
map("key", "kubernetes.io/cluster/${aws_eks_cluster.this.name}", "value", "owned", "propagate_at_launch", true),
map("key", "k8s.io/cluster-autoscaler/${lookup(var.worker_groups_launch_template[count.index], "autoscaling_enabled", local.workers_group_launch_template_defaults["autoscaling_enabled"]) == 1 ? "enabled" : "disabled" }", "value", "true", "propagate_at_launch", false)
),
local.asg_tags)
}"]
lifecycle {
ignore_changes = ["desired_capacity"]
}
}
resource "aws_launch_template" "workers_launch_template" {
name_prefix = "${aws_eks_cluster.this.name}-${lookup(var.worker_groups_launch_template[count.index], "name", count.index)}"
network_interfaces {
associate_public_ip_address = "${lookup(var.worker_groups_launch_template[count.index], "public_ip", local.workers_group_launch_template_defaults["public_ip"])}"
security_groups = ["${local.worker_security_group_id}", "${var.worker_additional_security_group_ids}", "${compact(split(",",lookup(var.worker_groups_launch_template[count.index],"additional_security_group_ids", local.workers_group_launch_template_defaults["additional_security_group_ids"])))}"]
}
iam_instance_profile = {
name = "${element(aws_iam_instance_profile.workers_launch_template.*.name, count.index)}"
}
image_id = "${lookup(var.worker_groups_launch_template[count.index], "ami_id", local.workers_group_launch_template_defaults["ami_id"])}"
instance_type = "${lookup(var.worker_groups_launch_template[count.index], "instance_type", local.workers_group_launch_template_defaults["instance_type"])}"
key_name = "${lookup(var.worker_groups_launch_template[count.index], "key_name", local.workers_group_launch_template_defaults["key_name"])}"
user_data = "${base64encode(element(data.template_file.launch_template_userdata.*.rendered, count.index))}"
ebs_optimized = "${lookup(var.worker_groups_launch_template[count.index], "ebs_optimized", lookup(local.ebs_optimized, lookup(var.worker_groups_launch_template[count.index], "instance_type", local.workers_group_launch_template_defaults["instance_type"]), false))}"
monitoring {
enabled = "${lookup(var.worker_groups_launch_template[count.index], "enable_monitoring", local.workers_group_launch_template_defaults["enable_monitoring"])}"
}
placement {
tenancy = "${lookup(var.worker_groups_launch_template[count.index], "placement_tenancy", local.workers_group_launch_template_defaults["placement_tenancy"])}"
}
count = "${var.worker_group_launch_template_count}"
lifecycle {
create_before_destroy = true
}
block_device_mappings {
device_name = "${data.aws_ami.eks_worker.root_device_name}"
ebs {
volume_size = "${lookup(var.worker_groups_launch_template[count.index], "root_volume_size", local.workers_group_launch_template_defaults["root_volume_size"])}"
volume_type = "${lookup(var.worker_groups_launch_template[count.index], "root_volume_type", local.workers_group_launch_template_defaults["root_volume_type"])}"
iops = "${lookup(var.worker_groups_launch_template[count.index], "root_iops", local.workers_group_launch_template_defaults["root_iops"])}"
delete_on_termination = true
}
}
}
resource "aws_iam_instance_profile" "workers_launch_template" {
name_prefix = "${aws_eks_cluster.this.name}"
role = "${lookup(var.worker_groups_launch_template[count.index], "iam_role_id", lookup(local.workers_group_launch_template_defaults, "iam_role_id"))}"
count = "${var.worker_group_launch_template_count}"
}