How To Create Reusable Infrastructure with Terraform Modules and Templates

The author selected the Free and Open Source Fund to receive a donation as part of the Write for DOnations program.

Introduction

One of the main benefits of Infrastructure as Code (IAC) is reusing parts of the defined infrastructure. In Terraform, you can use modules to encapsulate logically connected components into one entity and customize them using input variables you define. By using modules to define your infrastructure at a high level, you can separate development, staging, and production environments by only passing in different values to the same modules, which minimizes code duplication and maximizes conciseness.

You are not limited to using only your custom modules. Terraform Registry is integrated into Terraform and lists modules and providers that you can incorporate in your project right away by defining them in the required_providers section. Referencing public modules can speed up your workflow and reduce code duplication. If you have a useful module and would like to share it with the world, you can look into publishing it on the Registry for other developers to use.

In this tutorial, we’ll consider some of the ways of defining and reusing code in Terraform projects. You’ll reference modules from the Terraform Registry, separate development and production environments using modules, learn about templates and how they are used, and how to specify resource dependencies explicitly using the depends_on meta argument.

Prerequisites

  • A DigitalOcean Personal Access Token, which you can create via the DigitalOcean control panel. You can find instructions to do that at: How to Generate a Personal Access Token.
  • Terraform installed on your local machine and a project set up with the DigitalOcean provider. Complete Step 1 and Step 2 of the How To Use Terraform with DigitalOcean tutorial and be sure to name the project folder terraform-reusability, instead of loadbalance. During Step 2, do not include the pvt_key variable and the SSH key resource.
  • The droplet-lb module available under modules in terraform-reusability. Follow the How to Build a Custom Module tutorial and work through it until the droplet-lb module is functionally complete. (That is, until the cd ../.. command in the Creating a Module section.)
  • Knowledge of Terraform project structuring approaches. For more information, see How To Structure a Terraform Project.
  • (Optional) Two separate domains whose nameservers are pointed to DigitalOcean at your registrar. Refer to the How To Point to DigitalOcean Nameservers From Common Domain Registrars tutorial to set this up. Note that you don’t need to do this if you don’t plan on deploying the project you’ll create through this tutorial.

Note: We have specifically tested this tutorial using Terraform 0.13.

Separating Development and Production Environments

In this section, you’ll use modules to achieve separation between your target deployment environments. You’ll arrange these according to the structure of a more complex project. You’ll first create a project with two modules, one of which will define the Droplets and Load Balancers, and the other one will set up the DNS domain records. After, you’ll write configuration for two different environments (dev and prod), which will call the same modules.

Creating the dns-records module

As part of the prerequisites, you have set up the project initially under terraform-reusability and created the droplet-lb module in its own subdirectory under modules. You’ll now set up the second module, called dns-records, containing variables, outputs, and resource definitions. Assuming you’re in terraform-reusability, create dns-records by running:

  • mkdir modules/dns-records

Navigate to it:

  • cd modules/dns-records

This module will comprise the definitions for your domain and the DNS records that you’ll later point to the Load Balancers. You’ll first define the variables, which will become inputs that this module will expose. You’ll store them in a file called variables.tf. Create it for editing:

  • nano variables.tf

Add the following variable definitions:

terraform-reusability/modules/dns-records/variables.tf

variable "domain_name" {} variable "ipv4_address" {} 

Save and close the file. You’ll now define the domain and the accompanying A and CNAME records in a file named records.tf. Create and open it for editing by running:

  • nano records.tf

Add the following resource definitions:

terraform-reusability/modules/dns-records/records.tf

resource "digitalocean_domain" "domain" {   name = var.domain_name }  resource "digitalocean_record" "domain_A" {   domain = digitalocean_domain.domain.name   type   = "A"   name   = "@"   value  = var.ipv4_address }  resource "digitalocean_record" "domain_CNAME" {   domain = digitalocean_domain.domain.name   type   = "CNAME"   name   = "www"   value  = var.ipv4_address } 

First, you define the domain in your DigitalOcean account for your domain name. The cloud will automatically add the three DigitalOcean nameservers as NS records. Then, you define an A record for your domain, routing it (the @ as value signifies the true domain name, without subdomains) to the IP address supplied as the variable ipv4_address. For the sake of completeness, the CNAME record that follows specifies that the www subdomain should also point to the same IP address. Save and close the file when you’re done.

Next, you’ll define the outputs for this module. The outputs will show the FQDN (fully qualified domain name) of the created records. Create and open outputs.tf for editing:

  • nano outputs.tf

Add the following lines:

terraform-reusability/modules/dns-records/outputs.tf

output "A_fqdn" {   value = digitalocean_record.domain_A.fqdn }  output "CNAME_fqdn" {   value = digitalocean_record.domain_CNAME.fqdn } 

Save and close the file when you’re done.

With the variables, DNS records, and outputs defined, the last thing you’ll need to specify are the provider requirements for this module. You’ll specify that the dns-records module requires the digitalocean provider in a file called provider.tf. Create and open it for editing:

  • nano provider.tf

Add the following lines:

terraform-reusability/modules/dns-records/provider.tf

terraform {   required_providers {     digitalocean = {       source = "digitalocean/digitalocean"     }   }   required_version = ">= 0.13" } 

When you’re done, save and close the file. The dns-records module now requires the digitalocean provider and is functionally complete.

Creating Different Environments

The following is the current structure of the terraform-reusability project:

terraform_reusability/ ├─ modules/ │  ├─ dns-records/ │  │  ├─ outputs.tf │  │  ├─ provider.tf │  │  ├─ records.tf │  │  ├─ variables.tf │  ├─ droplet-lb/ │  │  ├─ droplets.tf │  │  ├─ lb.tf │  │  ├─ outputs.tf │  │  ├─ provider.tf │  │  ├─ variables.tf ├─ main.tf ├─ provider.tf 

So far, you have two modules in your project: the one you just created (dns-records) and droplet-lb, which you created as part of the prerequisites.

To facilitate different environments, you’ll store the dev and prod environment config files under a directory called environments, which will reside in the root of the project. Both environments will call the same two modules, but with different parameter values. The advantage of this is when the modules change internally in the future, you’ll only need to update the values you are passing in.

First, navigate to the root of the project by running:

  • cd ../..

Then, create the dev and prod directories under environments at the same time:

  • mkdir -p environments/dev && mkdir environments/prod

The -p argument orders mkdir to create all directories in the given path.

Navigate to the dev directory, as you’ll first configure that environment:

  • cd environments/dev

You’ll store the code in a file named main.tf, so create it for editing:

  • nano main.tf

Add the following lines:

terraform-reusability/environments/dev/main.tf

module "droplets" {   source   = "../../modules/droplet-lb"    droplet_count = 2   group_name    = "dev" }  module "dns" {   source   = "../../modules/dns-records"    domain_name   = "your_dev_domain"   ipv4_address  = module.droplets.lb_ip } 

Here you call and configure the two modules, droplet-lb and dns-records, which will together result in the creation of two Droplets. They’re fronted by a Load Balancer; the DNS records for the supplied domain are set up to point to that Load Balancer. Remember to replace your_dev_domain with your desired domain name for the dev environment, then save and close the file.

Next, you’ll configure the DigitalOcean provider and create a variable for it to be able to accept the personal access token you’ve created as part of the prerequisites. Open a new file, called provider.tf, for editing:

  • nano provider.tf

Add the following lines:

terraform-reusability/environments/dev/provider.tf

terraform {   required_providers {     digitalocean = {       source = "digitalocean/digitalocean"       version = "1.22.2"     }   } }  variable "do_token" {}  provider "digitalocean" {   token = var.do_token } 

In this code, you require the digitalocean provider to be available and pass in the do_token variable to its instance. Save and close the file.

Initialize the configuration by running:

  • terraform init

You’ll receive the following output:

Output Initializing modules... - dns in ../../modules/dns-records - droplets in ../../modules/droplet-lb  Initializing the backend...  Initializing provider plugins... - Finding latest version of digitalocean/digitalocean... - Installing digitalocean/digitalocean v2.0.2... - Installed digitalocean/digitalocean v2.0.2 (signed by a HashiCorp partner, key ID F82037E524B9C0E8)  Partner and community providers are signed by their developers. If you'd like to know more about provider signing, you can read about it here: https://www.terraform.io/docs/plugins/signing.html  The following providers do not have any version constraints in configuration, so the latest version was installed.  To prevent automatic upgrades to new major versions that may contain breaking changes, we recommend adding version constraints in a required_providers block in your configuration, with the constraint strings suggested below.  * digitalocean/digitalocean: version = "~> 2.0.2"  Terraform has been successfully initialized!  You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work.  If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary. 

The configuration for the prod environment is similar. Navigate to its directory by running:

  • cd ../prod

Create and open main.tf for editing:

  • nano main.tf

Add the following lines:

terraform-reusability/environments/prod/main.tf

module "droplets" {   source   = "../../modules/droplet-lb"    droplet_count = 5   group_name    = "prod" }  module "dns" {   source   = "../../modules/dns-records"    domain_name   = "your_prod_domain"   ipv4_address  = module.droplets.lb_ip } 

The difference between this and your dev code is that there will be five Droplets deployed. Furthermore, the domain name, which you should replace with your prod domain name, will be different. Save and close the file when you’re done.

Then, copy over the provider configuration from dev:

  • cp ../dev/provider.tf .

Initialize this configuration as well:

  • terraform init

The output of this command will be the same as the previous time you ran it.

You can try planning the configuration to see what resources Terraform would create by running:

  • terraform plan -var "do_token=${DO_PAT}"

The output for prod will be the following:

Output... An execution plan has been generated and is shown below. Resource actions are indicated with the following symbols:   + create  Terraform will perform the following actions:    # module.dns.digitalocean_domain.domain will be created   + resource "digitalocean_domain" "domain" {       + id   = (known after apply)       + name = "your_prod_domain"       + urn  = (known after apply)     }    # module.dns.digitalocean_record.domain_A will be created   + resource "digitalocean_record" "domain_A" {       + domain = "your_prod_domain"       + fqdn   = (known after apply)       + id     = (known after apply)       + name   = "@"       + ttl    = (known after apply)       + type   = "A"       + value  = (known after apply)     }    # module.dns.digitalocean_record.domain_CNAME will be created   + resource "digitalocean_record" "domain_CNAME" {       + domain = "your_prod_domain"       + fqdn   = (known after apply)       + id     = (known after apply)       + name   = "www"       + ttl    = (known after apply)       + type   = "CNAME"       + value  = (known after apply)     }    # module.droplets.digitalocean_droplet.droplets[0] will be created   + resource "digitalocean_droplet" "droplets" { ...       + name                 = "prod-0" ...     }    # module.droplets.digitalocean_droplet.droplets[1] will be created   + resource "digitalocean_droplet" "droplets" { ...       + name                 = "prod-1" ...     }    # module.droplets.digitalocean_droplet.droplets[2] will be created   + resource "digitalocean_droplet" "droplets" { ...       + name                 = "prod-2" ...     }    # module.droplets.digitalocean_droplet.droplets[3] will be created   + resource "digitalocean_droplet" "droplets" { ...       + name                 = "prod-3" ...     }    # module.droplets.digitalocean_droplet.droplets[4] will be created   + resource "digitalocean_droplet" "droplets" { ...       + name                 = "prod-4" ...     }    # module.droplets.digitalocean_loadbalancer.www-lb will be created   + resource "digitalocean_loadbalancer" "www-lb" { ...       + name                     = "lb-prod" ...  Plan: 9 to add, 0 to change, 0 to destroy. ... 

This would deploy five Droplets with a Load Balancer. Also it would create the prod domain you specified with the two DNS records pointing to the Load Balancer. You can try planning the configuration for the dev environment as well—you’ll note that two Droplets would be planned for deployment.

Note: You can apply this configuration for the dev and prod environments with the following command:

  • terraform apply -var "do_token=${DO_PAT}"

The following demonstrates how you have structured this project:

terraform_reusability/ ├─ environments/ │  ├─ dev/ │  │  ├─ main.tf │  │  ├─ provider.tf │  ├─ prod/ │  │  ├─ main.tf │  │  ├─ provider.tf ├─ modules/ │  ├─ dns-records/ │  │  ├─ outputs.tf │  │  ├─ provider.tf │  │  ├─ records.tf │  │  ├─ variables.tf │  ├─ droplet-lb/ │  │  ├─ droplets.tf │  │  ├─ lb.tf │  │  ├─ outputs.tf │  │  ├─ provider.tf │  │  ├─ variables.tf ├─ main.tf ├─ provider.tf 

The addition is the environments directory, which holds the code for the dev and prod environments.

The benefit of this approach is that further changes to modules automatically propagate to all areas of your project. Barring any possible customizations to module inputs, this approach is not repetitive and promotes reusability as much as possible, even across deployment environments. Overall this reduces clutter and allows you to trace the modifications using a version-control system.

In the final two sections of this tutorial, you’ll review the depends_on meta argument and the templatefile function.

Declaring Dependencies to Build Infrastructure in Order

While planning actions, Terraform automatically tries to sense existing dependencies and builds them into its dependency graph. The main dependencies it can detect are clear references; for example, when an output value of a module is passed to a parameter on another resource. In this scenario the module must first complete its deployment to provide the output value.

The dependencies that Terraform can’t detect are hidden—they have side effects and mutual references not inferable from the code. An example of this is when an object depends not on the existence, but on the behavior of another one, and does not access its attributes from code. To overcome this, you can use depends_on to manually specify the dependencies in an explicit way. Since Terraform 0.13, you can also use depends_on on modules to force the listed resources to be fully deployed before deploying the module itself. It’s possible to use the depends_on meta argument with every resource type. depends_on will also accept a list of other resources on which its specified resource depends.

In the previous step of this tutorial, you haven’t specified any explicit dependencies using depends_on, because the resources you’ve created have no side effects not inferable from the code. Terraform is able to detect the references made from the code you’ve written, and will schedule the resources for deployment accordingly.

depends_on accepts a list of references to other resources. Its syntax looks like this:

resource "resource_type" "res" {   depends_on = [...] # List of resources    # Parameters... } 

Remember that you should only use depends_on as a last-resort option. If used, it should be kept well documented, because the behavior that the resources depend on may not be immediately obvious.

Using Templates for Customization

In Terraform, templating is substituting results of expressions in appropriate places, such as when setting attribute values on resources or constructing strings. You’ve used it in the previous steps and the tutorial prerequisites to dynamically generate Droplet names and other parameter values.

When substituting values in strings, the values are specified and surrounded by ${}. Template substitution is often used in loops to facilitate customization of the created resources. It also allows for module customization by substituting inputs in resource attributes.

Terraform offers the templatefile function, which accepts two arguments: the file from the disk to read and a map of variables paired with their values. The value it returns is the contents of the file rendered with the expression substituted—just as Terraform would normally do when planning or applying the project. Because functions are not part of the dependency graph, the file cannot be dynamically generated from another part of the project.

Imagine that the contents of the template file called droplets.tmpl is as follows:

%{ for address in addresses ~} ${address}:80 %{ endfor ~} 

Longer declarations must be surrounded with %{}, as is the case with the for and endfor declarations, which signify the start and end of the for loop respectively. The contents and type of the droplets variable are not known until the function is called and actual values provided, like so:

templatefile("${path.module}/droplets.tmpl", { addresses = ["192.168.0.1", "192.168.1.1"] }) 

The value that this templatefile call will return is the following:

Output192.168.0.1:80 192.168.1.1:80 

This function has its use cases, but they are uncommon. For example, you could use it when a part of the configuration is necessary to exist in a proprietary format, but is dependent on the rest of the values and must be generated dynamically. In the majority of cases, it’s better to specify all configuration parameters directly in Terraform code, where possible.

Conclusion

In this article, you’ve maximized code reuse in an example Terraform project. The main way is to package often-used features and configurations as a customizable module and use it whenever needed. By doing so, you do not duplicate the underlying code (which can be error prone) and enable faster turnaround times, since modifying the module is almost all you need to do to introduce changes.

You’re not limited to your own modules. As you’ve seen, Terraform Registry provides third-party modules and providers that you can incorporate in your project.

Check out the rest of the How To Manage Infrastructure with Terraform series.