example-dhall-terraform-github

Example of using Dhall to generate terraform file to manage a Github Organisation

What is Terraform?

HashiCorp Terraform enables you to safely and predictably create, change, and improve infrastructure. It is an open source tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.

https://www.terraform.io/

Terraform with the Github Provider

The GitHub provider is used to interact with GitHub organization resources. The provider allows you to manage your GitHub organization's members and teams easily.

https://www.terraform.io/docs/providers/github/index.html

What we would like to achieve?

We would like to represent the users teams, and memberships of our github organisation, in a code as configuration way. We will use this code to add and remove users in the future.

Why terraform is not enough?

Plain old terraform

We could write plain old terraform.

resource "github_membership" "memberships_user1" {	
  username = "user1"	
  role     = "member"	
}

resource "github_membership" "memberships_user2" {	
  username = "user2"	
  role     = "member"	
}

resource "github_team" "some_team" {
  name        = "SomeTeam"
  description = "Some cool team"
}

resource "github_team_membership" "some_team_membership" {
  team_id  = "${github_team.some_team.id}"
  username = "user1"
  role     = "member"
}

But if you have 10s or 100s of users in your organisation, with teams, you will have a very verbose codebase.

Terraform with array and map

Managing Terraform users and membership in a few lines of code would make it easy to have a global view. So having a configuration variable like this on below would be ideal.

variable "organisation_memberships" {	
  type = "list"	
   # written here because multiple map declarations is not supported in terraform.tfvars.json	
  default = [	
    {	
      login = "user1"	
      role = "admin"	
      teams = "all,some_team"	
    },	
    {	
      login = "user2"	
      teams = "all,some_team"	
    }
  ]
}

resource "github_membership" "memberships" {	
  count = "${length(var.organisation_memberships)}"	
  username = "${lookup(var.organisation_memberships[count.index], "login")}"	
  role     = "${lookup(var.organisation_memberships[count.index], "role", "member")}"	
} 

However this solution has a major drawback. Terraform use the index position of the array when generating the resources (github_membership.memberships.1). If you need to add or remove a user in this array, this would lead to most of the resources being deleted and then recreated. For example, adding a user at the beginning, would mean the user membership at the index 1 would be deleted and recreated at the index 2, and this is valid for the subsequent users. These deletions and creations would trigger emails invitation to join the github organisation for every users...

If only there was a way to not have static name, and be able to generate something like github_membership.memberships.user_1 From there we could generate terraform with terraform, or knowing the limitation of Terraform pick a language a bit more powerful.

Dhall

What is Dhall?

Dhall is a programmable configuration language that is not Turing-complete You can think of Dhall as: JSON + functions + types + imports

https://github.com/dhall-lang/dhall-lang

So we have decided to pick up Dhall.

Repo structure

The Dhall folder is organised this way:

• dhall/config.dhall for the config.
• dhall/config.dhall for the defaults values (helps keep the config file short)
• dhall/config.dhall for the Types (Types are values in Dhall)
• dhall/config.dhall for the Github / Terraform specific code.

The file dhall.tf.json is the reprensation of the of our Dhall config in Terraform world. It is generated by running

dhall-to-json --pretty <<< './dhall/config.dhall' > ./dhall.tf.json 

In the future, we could refactor our Dhall codebase to only allow valid terraform to be produced.

How to update teams and users?

1. Update dhall config

Update file dhall/config.dhall

2. Generate dhall.tf.json

Once you have installed dhall, you should have access to dhall-to-json

dhall-to-json --pretty <<< './dhall/config.dhall' > ./dhall.tf.json 

This file is a json representation of all the resources terraform will manage for us.

3. Run terraform plan

4. Run terraform apply

Links

https://github.com/blast-hardcheese/dhall-terraform