Merge branch 'release/4.15.0'

CHANGELOG.md

@@ -7,11 +7,19 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
 
 ## Unreleased
 
-## 4.14.0 - 2020-04094
+
+## 4.15.0 - 2020-04-16
+
+- Added: support custom docker machine distribution (#216) …
+- Bugfix: disabled cache (#212) 
+- Bugfix: failing curl (#217) …
+- Change: Drop supported to manage ec2 keys (#192)
+
+
+## 4.14.0 - 2020-04-04
 - Add: Allow traffic from a list of security group IDs (#207) by @fliphess
 - Bugfix: Fix missing policy for existing cache (#208, #206)
 
-
 ## 4.13.0 - 2020-03-26
 - Add: variables `cache_lifecycle_prefix` and `cache_lifecycle_clear` to increase flexibility of the cache usages.
 - Add: Parametrize the AWS ARN for policies (#203) @ericamador
@@ -296,8 +304,9 @@ Module is available as Terraform 0.11 module, pin module to version 3.x. Please
 - Update default AMI's to The latest Amazon Linux AMI 2017.09.1 - released on 2018-01-17.
 - Minor updates in the example
 
-[Unreleased]: https://github.com/npalm/terraform-aws-gitlab-runner/compare/4.14.0...HEAD
-[4.13.0]: https://github.com/npalm/terraform-aws-gitlab-runner/compare/4.14.0...4.13.0
+[Unreleased]: https://github.com/npalm/terraform-aws-gitlab-runner/compare/4.15.0...HEAD
+[4.15.0]: https://github.com/npalm/terraform-aws-gitlab-runner/compare/4.15.0...4.14.0
+[4.14.0]: https://github.com/npalm/terraform-aws-gitlab-runner/compare/4.14.0...4.13.0
 [4.13.0]: https://github.com/npalm/terraform-aws-gitlab-runner/compare/4.13.0...4.12.0
 [4.12.0]: https://github.com/npalm/terraform-aws-gitlab-runner/compare/4.12.0...4.11.1
 [4.11.1]: https://github.com/npalm/terraform-aws-gitlab-runner/compare/4.11.1...4.11.0

README.md

@@ -2,6 +2,10 @@
 
 # Terraform module for GitLab auto scaling runners on AWS spot instances
 
+> "Added support to download docker machine from a different location, e.g. <https://gitlab.com/gitlab-org/ci-cd/docker-machine>"
+
+> "Managed ec2 key support dropped": The module will not longer manage an SSH key pair. The module offers two way to access instances. First via the AWS session manager and second by providing an AWS key pair as parameter.
+
 > "Type changes": The types of variable `runners_volumes_tmpfs`, and `runners_services_volumes_tmpfs` are changed to support the Terraform 12 `templatefile` function. Check the [default example](examples/runner-pre-registered/main.tf) for an usages example.
 
 ## Terraform versions
@@ -257,11 +261,12 @@ terraform destroy
 | cache\_expiration\_days | Number of days before cache objects expires. | `number` | `1` | no |
 | cache\_shared | Enables cache sharing between runners, false by default. | `bool` | `false` | no |
 | cloudwatch\_logging\_retention\_in\_days | Retention for cloudwatch logs. Defaults to unlimited | `number` | `0` | no |
-| docker\_machine\_instance\_type | Instance type used for the instances hosting docker-machine. | `string` | `"m5a.large"` | no |
+| docker\_machine\_download\_url | Full url pointing to a linux x64 distribution of docker machine. Once set `docker_machine_version` will be ingored. For example the GitLab version, https://gitlab-docker-machine-downloads.s3.amazonaws.com/v0.16.2-gitlab.2/docker-machine. | `string` | `""` | no |
+| docker\_machine\_instance\_type | Instance type used for the instances hosting docker-machine. | `string` | `"m5.large"` | no |
 | docker\_machine\_options | List of additional options for the docker machine config. Each element of this list must be a key=value pair. E.g. '["amazonec2-zone=a"]' | `list(string)` | `[]` | no |
 | docker\_machine\_role\_json | Docker machine runner instance override policy, expected to be in JSON format. | `string` | `""` | no |
 | docker\_machine\_spot\_price\_bid | Spot price bid. | `string` | `"0.06"` | no |
-| docker\_machine\_version | Version of docker-machine. | `string` | `"0.16.2"` | no |
+| docker\_machine\_version | Version of docker-machine. The version will be ingored once `docker_machine_download_url` is set. | `string` | `"0.16.2"` | no |
 | enable\_cloudwatch\_logging | Boolean used to enable or disable the CloudWatch logging. | `bool` | `true` | no |
 | enable\_eip | Enable the assignment of an EIP to the gitlab runner instance | `bool` | `false` | no |
 | enable\_forced\_updates | Enable automatic redeployment of the Runner ASG when the Launch Configs change. | `bool` | `false` | no |
@@ -274,7 +279,8 @@ terraform destroy
 | enable\_schedule | Flag used to enable/disable auto scaling group schedule for the runner instance. | `bool` | `false` | no |
 | environment | A name that identifies the environment, used as prefix and for tagging. | `string` | n/a | yes |
 | gitlab\_runner\_registration\_config | Configuration used to register the runner. See the README for an example, or reference the examples in the examples directory of this repo. | `map(string)` | <pre>{<br>  "access_level": "",<br>  "description": "",<br>  "locked_to_project": "",<br>  "maximum_timeout": "",<br>  "registration_token": "",<br>  "run_untagged": "",<br>  "tag_list": ""<br>}</pre> | no |
-| gitlab\_runner\_ssh\_cidr\_blocks | List of CIDR blocks to allow SSH Access to the gitlab runner instance. | `list(string)` | <pre>[<br>  "0.0.0.0/0"<br>]</pre> | no |
+| gitlab\_runner\_security\_group\_ids | A list of security group ids that are allowed to access the gitlab runner agent | `list(string)` | `[]` | no |
+| gitlab\_runner\_ssh\_cidr\_blocks | List of CIDR blocks to allow SSH Access to the gitlab runner instance. | `list(string)` | `[]` | no |
 | gitlab\_runner\_version | Version of the GitLab runner. | `string` | `"12.8.0"` | no |
 | instance\_role\_json | Default runner instance override policy, expected to be in JSON format. | `string` | `""` | no |
 | instance\_type | Instance type used for the GitLab runner. | `string` | `"t3.micro"` | no |
@@ -323,8 +329,7 @@ terraform destroy
 | runners\_volumes\_tmpfs | n/a | <pre>list(object({<br>    volume  = string<br>    options = string<br>  }))</pre> | `[]` | no |
 | schedule\_config | Map containing the configuration of the ASG scale-in and scale-up for the runner instance. Will only be used if enable\_schedule is set to true. | `map` | <pre>{<br>  "scale_in_count": 0,<br>  "scale_in_recurrence": "0 18 * * 1-5",<br>  "scale_out_count": 1,<br>  "scale_out_recurrence": "0 8 * * 1-5"<br>}</pre> | no |
 | secure\_parameter\_store\_runner\_token\_key | The key name used store the Gitlab runner token in Secure Parameter Store | `string` | `"runner-token"` | no |
-| ssh\_key\_pair | Set this to use existing AWS key pair | `string` | `""` | no |
-| ssh\_public\_key | Public SSH key used for the GitLab runner EC2 instance. | `string` | `""` | no |
+| ssh\_key\_pair | Set this to use existing AWS key pair | `string` | n/a | yes |
 | subnet\_id\_runners | List of subnets used for hosting the gitlab-runners. | `string` | n/a | yes |
 | subnet\_ids\_gitlab\_runner | Subnet used for hosting the GitLab runner. | `list(string)` | n/a | yes |
 | tags | Map of tags that will be added to created resources. By default resources will be tagged with name and environment. | `map(string)` | `{}` | no |

_docs/README.md

@@ -3,6 +3,10 @@
 
 # Terraform module for GitLab auto scaling runners on AWS spot instances
 
+> "Added support to download docker machine from a different location, e.g. https://gitlab.com/gitlab-org/ci-cd/docker-machine"
+
+> "Managed ec2 key support dropped": The module will not longer manage an SSH key pair. The module offers two way to access instances. First via the AWS session manager and second by providing an AWS key pair as parameter. 
+
 > "Type changes": The types of variable `runners_volumes_tmpfs`, and  `runners_services_volumes_tmpfs` are changed to support the Terraform 12 `templatefile` function. Check the [default example](examples/runner-pre-registered/main.tf) for an usages example.
 
 ## Terraform versions

_docs/TF_MODULE.md

@@ -23,11 +23,12 @@
 | cache\_expiration\_days | Number of days before cache objects expires. | `number` | `1` | no |
 | cache\_shared | Enables cache sharing between runners, false by default. | `bool` | `false` | no |
 | cloudwatch\_logging\_retention\_in\_days | Retention for cloudwatch logs. Defaults to unlimited | `number` | `0` | no |
-| docker\_machine\_instance\_type | Instance type used for the instances hosting docker-machine. | `string` | `"m5a.large"` | no |
+| docker\_machine\_download\_url | Full url pointing to a linux x64 distribution of docker machine. Once set `docker_machine_version` will be ingored. For example the GitLab version, https://gitlab-docker-machine-downloads.s3.amazonaws.com/v0.16.2-gitlab.2/docker-machine. | `string` | `""` | no |
+| docker\_machine\_instance\_type | Instance type used for the instances hosting docker-machine. | `string` | `"m5.large"` | no |
 | docker\_machine\_options | List of additional options for the docker machine config. Each element of this list must be a key=value pair. E.g. '["amazonec2-zone=a"]' | `list(string)` | `[]` | no |
 | docker\_machine\_role\_json | Docker machine runner instance override policy, expected to be in JSON format. | `string` | `""` | no |
 | docker\_machine\_spot\_price\_bid | Spot price bid. | `string` | `"0.06"` | no |
-| docker\_machine\_version | Version of docker-machine. | `string` | `"0.16.2"` | no |
+| docker\_machine\_version | Version of docker-machine. The version will be ingored once `docker_machine_download_url` is set. | `string` | `"0.16.2"` | no |
 | enable\_cloudwatch\_logging | Boolean used to enable or disable the CloudWatch logging. | `bool` | `true` | no |
 | enable\_eip | Enable the assignment of an EIP to the gitlab runner instance | `bool` | `false` | no |
 | enable\_forced\_updates | Enable automatic redeployment of the Runner ASG when the Launch Configs change. | `bool` | `false` | no |
@@ -40,8 +41,8 @@
 | enable\_schedule | Flag used to enable/disable auto scaling group schedule for the runner instance. | `bool` | `false` | no |
 | environment | A name that identifies the environment, used as prefix and for tagging. | `string` | n/a | yes |
 | gitlab\_runner\_registration\_config | Configuration used to register the runner. See the README for an example, or reference the examples in the examples directory of this repo. | `map(string)` | <pre>{<br>  "access_level": "",<br>  "description": "",<br>  "locked_to_project": "",<br>  "maximum_timeout": "",<br>  "registration_token": "",<br>  "run_untagged": "",<br>  "tag_list": ""<br>}</pre> | no |
-| gitlab\_runner\_ssh\_cidr\_blocks | List of CIDR blocks to allow SSH Access to the gitlab runner instance. | `list(string)` | <pre>[<br>]</pre> | no |
-| gitlab\_runner\_security\_group\_ids | List of security group IDs to allow Access to the gitlab runner instances. | `list(string)` | <pre>[<br>]</pre>` | no |
+| gitlab\_runner\_security\_group\_ids | A list of security group ids that are allowed to access the gitlab runner agent | `list(string)` | `[]` | no |
+| gitlab\_runner\_ssh\_cidr\_blocks | List of CIDR blocks to allow SSH Access to the gitlab runner instance. | `list(string)` | `[]` | no |
 | gitlab\_runner\_version | Version of the GitLab runner. | `string` | `"12.8.0"` | no |
 | instance\_role\_json | Default runner instance override policy, expected to be in JSON format. | `string` | `""` | no |
 | instance\_type | Instance type used for the GitLab runner. | `string` | `"t3.micro"` | no |
@@ -90,8 +91,7 @@
 | runners\_volumes\_tmpfs | n/a | <pre>list(object({<br>    volume  = string<br>    options = string<br>  }))</pre> | `[]` | no |
 | schedule\_config | Map containing the configuration of the ASG scale-in and scale-up for the runner instance. Will only be used if enable\_schedule is set to true. | `map` | <pre>{<br>  "scale_in_count": 0,<br>  "scale_in_recurrence": "0 18 * * 1-5",<br>  "scale_out_count": 1,<br>  "scale_out_recurrence": "0 8 * * 1-5"<br>}</pre> | no |
 | secure\_parameter\_store\_runner\_token\_key | The key name used store the Gitlab runner token in Secure Parameter Store | `string` | `"runner-token"` | no |
-| ssh\_key\_pair | Set this to use existing AWS key pair | `string` | `""` | no |
-| ssh\_public\_key | Public SSH key used for the GitLab runner EC2 instance. | `string` | `""` | no |
+| ssh\_key\_pair | Set this to use existing AWS key pair | `string` | n/a | yes |
 | subnet\_id\_runners | List of subnets used for hosting the gitlab-runners. | `string` | n/a | yes |
 | subnet\_ids\_gitlab\_runner | Subnet used for hosting the GitLab runner. | `list(string)` | n/a | yes |
 | tags | Map of tags that will be added to created resources. By default resources will be tagged with name and environment. | `map(string)` | `{}` | no |

ci/bin/terraform.sh

@@ -3,17 +3,16 @@
 TARGET_DIR=/opt
 PATH=${PATH}:${TARGET_DIR}
 
-TERRAFORM_VERSION=${1:-"0.12.8"}
+TERRAFORM_VERSION=${1:-"0.12.24"}
 OS=${2:-"linux"}
 TERRAFORM_URL="https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_${OS}_amd64.zip"
 
 installTerraform() {
   echo "Downloading terraform: ${TERRAFORM_URL}"
 
-  curl '-#' -fL -o ${TARGET_DIR}/terraform.zip ${TERRAFORM_URL} && \
-    unzip -q -d ${TARGET_DIR}/ ${TARGET_DIR}/terraform.zip && \
-
-  terraform --version
+  curl '-#' -fL -o ${TARGET_DIR}/terraform.zip ${TERRAFORM_URL} &&
+    unzip -q -d ${TARGET_DIR}/ ${TARGET_DIR}/terraform.zip &&
+    terraform --version
 }
 
 verifyModulesAndPlugins() {
@@ -23,7 +22,7 @@ verifyModulesAndPlugins() {
 
 formatCheck() {
   RESULT=$(terraform fmt -recursive -write=false)
-  if [[ ! -z ${RESULT} ]] ; then
+  if [[ ! -z ${RESULT} ]]; then
     echo The following files are formatted incorrectly: $RESULT
     exit 1
   fi
@@ -33,4 +32,3 @@ validate() {
   echo "Validating and checking format of terraform code in $PWD"
   terraform validate
 }
-

examples/runner-default/.terraform-version

@@ -1 +1 @@
-0.12.23
+0.12.24

examples/runner-default/README.md

@@ -8,6 +8,7 @@ This examples shows:
   - No SSH keys, you can log into the instance via SSM (Session Manager).
   - Registration via GitLab token.
   - Auto scaling using `docker+machine` executor.
+  - Addtional security groups that are allowed access to the runner agent
 
 ![runners-default](https://github.com/npalm/assets/raw/master/images/terraform-aws-gitlab-runner/runner-default.png)
 
@@ -19,7 +20,7 @@ The Terraform version is managed using [tfenv](https://github.com/Zordrak/tfenv)
 
 | Name | Version |
 |------|---------|
-| aws | 2.41 |
+| aws | 2.56 |
 | null | 2.1.2 |
 
 ## Inputs

examples/runner-default/_docs/TF_MODULE.md

@@ -2,7 +2,7 @@
 
 | Name | Version |
 |------|---------|
-| aws | 2.41 |
+| aws | 2.56 |
 | null | 2.1.2 |
 
 ## Inputs

examples/runner-default/main.tf

@@ -3,12 +3,13 @@ data "aws_availability_zones" "available" {
 }
 
 data "aws_security_group" "default" {
-  name = "default"
+  name   = "default"
+  vpc_id = module.vpc.vpc_id
 }
 
 module "vpc" {
   source  = "terraform-aws-modules/vpc/aws"
-  version = "2.21"
+  version = "2.33"
 
   name = "vpc-${var.environment}"
   cidr = "10.0.0.0/16"
@@ -43,6 +44,7 @@ module "runner" {
 
   gitlab_runner_security_group_ids = [data.aws_security_group.default.id]
 
+  docker_machine_download_url   = "https://gitlab-docker-machine-downloads.s3.amazonaws.com/v0.16.2-gitlab.2/docker-machine"
   docker_machine_spot_price_bid = "0.06"
 
   gitlab_runner_registration_config = {

examples/runner-default/providers.tf

@@ -1,6 +1,6 @@
 provider "aws" {
   region  = var.aws_region
-  version = "2.41"
+  version = "2.56"
 }
 
 provider "local" {

examples/runner-docker/.terraform-version

@@ -1 +1 @@
-0.12.23
+0.12.24

examples/runner-docker/README.md

@@ -25,10 +25,7 @@ The terraform version is managed using [tfenv](https://github.com/Zordrak/tfenv)
 
 | Name | Version |
 |------|---------|
-| aws | 2.41 |
-| local | 1.4 |
-| null | 2.1.2 |
-| tls | 2.1.1 |
+| aws | 2.56 |
 
 ## Inputs
 
@@ -37,8 +34,6 @@ The terraform version is managed using [tfenv](https://github.com/Zordrak/tfenv)
 | aws\_region | AWS region. | `string` | `"eu-west-1"` | no |
 | environment | A name that identifies the environment, will used as prefix and for tagging. | `string` | `"runners-docker"` | no |
 | gitlab\_url | URL of the gitlab instance to connect to. | `string` | `"https://gitlab.com"` | no |
-| private\_ssh\_key\_filename | n/a | `string` | `"generated/id_rsa"` | no |
-| public\_ssh\_key\_filename | n/a | `string` | `"generated/id_rsa.pub"` | no |
 | registration\_token | n/a | `any` | n/a | yes |
 | runner\_name | Name of the runner, will be used in the runner config.toml | `string` | `"docker"` | no |
 

examples/runner-docker/_docs/TF_MODULE.md

@@ -2,10 +2,7 @@
 
 | Name | Version |
 |------|---------|
-| aws | 2.41 |
-| local | 1.4 |
-| null | 2.1.2 |
-| tls | 2.1.1 |
+| aws | 2.56 |
 
 ## Inputs
 
@@ -14,8 +11,6 @@
 | aws\_region | AWS region. | `string` | `"eu-west-1"` | no |
 | environment | A name that identifies the environment, will used as prefix and for tagging. | `string` | `"runners-docker"` | no |
 | gitlab\_url | URL of the gitlab instance to connect to. | `string` | `"https://gitlab.com"` | no |
-| private\_ssh\_key\_filename | n/a | `string` | `"generated/id_rsa"` | no |
-| public\_ssh\_key\_filename | n/a | `string` | `"generated/id_rsa.pub"` | no |
 | registration\_token | n/a | `any` | n/a | yes |
 | runner\_name | Name of the runner, will be used in the runner config.toml | `string` | `"docker"` | no |
 

examples/runner-docker/main.tf

@@ -4,7 +4,7 @@ data "aws_availability_zones" "available" {
 
 module "vpc" {
   source  = "terraform-aws-modules/vpc/aws"
-  version = "2.21"
+  version = "2.33"
 
   name = "vpc-${var.environment}"
   cidr = "10.1.0.0/16"
@@ -19,13 +19,20 @@ module "vpc" {
   }
 }
 
+module "key_pair" {
+  source = "../../modules/key-pair"
+
+  environment = var.environment
+  name        = var.runner_name
+}
+
 module "runner" {
   source = "../../"
 
   aws_region  = var.aws_region
   environment = var.environment
 
-  ssh_public_key = local_file.public_ssh_key.content
+  ssh_key_pair = module.key_pair.key_pair.key_name
 
   runners_use_private_address = false
   enable_eip                  = true

examples/runner-docker/providers.tf

@@ -1,6 +1,6 @@
 provider "aws" {
   region  = var.aws_region
-  version = "2.41"
+  version = "2.56"
 }
 
 provider "local" {

examples/runner-docker/variables.tf

@@ -10,13 +10,13 @@ variable "environment" {
   type        = string
 }
 
-variable "public_ssh_key_filename" {
-  default = "generated/id_rsa.pub"
-}
+# variable "public_ssh_key_filename" {
+#   default = "generated/id_rsa.pub"
+# }
 
-variable "private_ssh_key_filename" {
-  default = "generated/id_rsa"
-}
+# variable "private_ssh_key_filename" {
+#   default = "generated/id_rsa"
+# }
 
 variable "runner_name" {
   description = "Name of the runner, will be used in the runner config.toml"

examples/runner-pre-registered/.terraform-version

@@ -1 +1 @@
-0.12.23
+0.12.24

examples/runner-pre-registered/README.md

@@ -10,10 +10,7 @@ The terraform version is managed using [tfenv](https://github.com/Zordrak/tfenv)
 
 | Name | Version |
 |------|---------|
-| aws | 2.41 |
-| local | 1.4 |
-| null | 2.1.2 |
-| tls | 2.1.1 |
+| aws | 2.52 |
 
 ## Inputs
 

examples/runner-pre-registered/_docs/TF_MODULE.md

@@ -2,10 +2,7 @@
 
 | Name | Version |
 |------|---------|
-| aws | 2.41 |
-| local | 1.4 |
-| null | 2.1.2 |
-| tls | 2.1.1 |
+| aws | 2.52 |
 
 ## Inputs