How To Troubleshoot Terraform

The author selected the Free and Open Source Fund to receive a donation as part of the Write for DOnations program.

Introduction

Things do not always go according to plan: deployments can fail, existing resources may break unexpectedly, and you and your team could be forced to fix the issue as soon as possible. Understanding the methods to approach debugging your Terraform project is crucial when you need to make a swift response.

Similarly to developing with other programming languages and frameworks, setting log levels in Terraform to gain insight into its internal workflows with the necessary verbosity is a feature that can help you when troubleshooting. By logging the internal actions, you can uncover implicitly hidden errors, such as variables defaulting to an unsuitable data type. Also common with frameworks is the ability to import and use third-party modules (or libraries) to reduce code duplication between projects.

In this tutorial, you’ll verify that variables always have sensible values and you’ll specify exactly which versions of providers and modules you need to prevent conflicts. You’ll also enable various levels of debug mode verbosity, which can help you diagnose an underlying issue in Terraform itself.

Prerequisites

• A DigitalOcean Personal Access Token, which you can create via the DigitalOcean Control Panel. You can find instructions to do this in How to Generate a Personal Access Token.
• Terraform installed on your local machine and a project set up with the DigitalOcean provider. Complete Step 1 and Step 2 of the How To Use Terraform with DigitalOcean tutorial, and be sure to name the project folder terraform-troubleshooting, instead of loadbalance. During Step 2, do not include the pvt_key variable and the SSH key resource.

Note: We have specifically tested this tutorial using Terraform 0.13.

Setting Version Constraints

Although the ability to make use of third-party modules and providers can minimize code duplication and effort when writing your Terraform code, it is also possible for developers of third-party modules to release new versions that can potentially bring breaking changes to your specific code. To prevent this, Terraform allows you to specify version boundaries to ensure that only the versions you want are installed and used. Specifying versions means that you will require the versions you’ve tested in your code, but it also leaves the possibility for a future update.

Version constraints are specified as strings and passed in to the version parameter when you define module or provider requirements. As part of the Prerequisites, you’ve already requested the digitalocean provider in provider.tf. Run the following command to review what version requirements it specifies:

cat provider.tf

You’ll find the provider code as follows:

terraform {
  required_providers {
    digitalocean = {
      source = "digitalocean/digitalocean"
      version = "1.22.2"
    }
  }
}
...

In this case, you have requested the digitalocean provider, and explicitly set the version to 1.22.2. When your project only requires a specific version, this is the most effortless way to accomplish that.

Version constraints can be more complex than just specifying one version. They can contain one or more groups of conditions, separated by a comma (,). The groups each define an acceptable range of versions and may include operators, such as:

• >, <, >=, <=: for comparisons, such as >=1.0, which would require the version to be equal to or greater than 1.0.
• !=: for excluding a specific version—!= 1.0 would deny version 1.0 from being used and requested.
• ~>: for matching the specified version up to the right-most version part, which is allowed to increment (~>1.5.10 will match 1.5.10 and 1.5.11, but won’t match 1.5.9).

Here are two examples of version constraints with multiple groups:

• >=1.0, <2.0: allows all versions from the 1.0 series onward, up to 2.0.
• >1.0, != 1.5: allows versions greater than, but not equal to 1.0, with the exception of 1.5, which it also excludes.

For a potential available version to be selected, it must pass every specified constraint and remain compatible with other modules and providers, as well as the version of Terraform that you’re using. If Terraform deems no combination acceptable, it won’t be able to perform any tasks because the dependencies remain unresolved. When Terraform identifies acceptable versions satisfying the constraints, it uses the latest one available.

In this section, you’ve learned about locking the range of module and resource versions that you can install in your project by specifying version constraints. This is useful when you want stability by using only tested and approved versions of third-party code. In the next section, you’ll configure Terraform to show more verbose logs, which are necessary for bug reports and further debugging in case of crashes.

Enabling Debug Mode

There could be a bug or malformed input within your workflow, which may result in your resources not provisioning as intended. In such rare cases, it’s important to know how to access detailed logs describing what Terraform is doing. They may aid in pinpointing the cause of the error, tell you if it’s user-made, or prompt you to report the issue to Terraform developers if it’s an internal bug.

Terraform exposes the TF_LOG environment variable for setting the level of logging verbosity, of which there are five:

• TRACE: the most elaborate verbosity, shows every step taken by Terraform and produces enormous outputs with internal logs.
• DEBUG: describes what happens internally in a more concise way compared to TRACE.
• ERROR: shows errors that prevent Terraform from continuing.
• WARN: logs warnings, which may indicate misconfiguration or mistakes, but are not critical to execution.
• INFO: shows general, high-level messages about the execution process.

To specify a desired log level, you’ll have to set the environment variable to the appropriate value:

export TF_LOG=log_level

If TF_LOG is defined, but the value is not one of the five listed verbosity levels, Terraform will default to TRACE.

You’ll now define a Droplet resource and try deploying it with different log levels. You’ll store the Droplet definition in a file named droplets.tf, so create and open it for editing:

nano droplets.tf

Add the following lines:

resource "digitalocean_droplet" "test-droplet" {
  image  = "ubuntu-18-04-x64"
  name   = "test-droplet"
  region = "fra1"
  size   = "s-1vcpu-1gb"
}

This Droplet will run Ubuntu 18.04 with one CPU core and 1GB RAM in the fra1 region; you’ll call it test-droplet. That is all you need to define, so save and close the file.

Before deploying the Droplet, set the log level to DEBUG by running:

export TF_LOG=DEBUG

Then, plan the Droplet provisioning:

terraform plan -var "do_token=${DO_PAT}"

The output will be very long, and you can inspect it more closely to find that each line starts with the level of verbosity (importance) in double brackets. You’ll see that most of the lines start with [DEBUG].

[WARN] and [INFO] are also present; that’s because TF_LOG sets the lowest log level. This means that you’d have to set TF_LOG to TRACE to show TRACE and all other log levels at the same time.

If an internal error occurred, Terraform will show the stack trace and exit, stopping execution. From there, you’ll be able to locate where in the source code the error occurred, and if it’s a bug, report it to Terraform developers. Otherwise, if it’s an error in your code, Terraform will point it out to you, so you can fix it in your project.

Here is how the log output would be when the DigitalOcean backend can’t verify your API token. It throws a user error because of incorrect input:

Output
...
digitalocean_droplet.test-droplet: Creating...
2021/01/20 06:54:35 [ERROR] eval: *terraform.EvalApplyPost, err: Error creating droplet: POST https://api.digitalocean.com/v2/droplets: 401 Unable to authenticate you
2021/01/20 06:54:35 [ERROR] eval: *terraform.EvalSequence, err: Error creating droplet: POST https://api.digitalocean.com/v2/droplets: 401 Unable to authenticate you

Error: Error creating droplet: POST https://api.digitalocean.com/v2/droplets: 401 Unable to authenticate you

  on droplets.tf line 1, in resource "digitalocean_droplet" "test-droplet":
   1: resource "digitalocean_droplet" "test-droplet" {
   ...

You’ve now learned to enable more verbose logging modes. They are very useful for diagnosing crashes and unexpected Terraform behavior. In the next section, you’ll review verifying variables and preventing edge cases.

Validating Variables

In this section, you’ll ensure that variables always have sensible and appropriate values according to their type and validation parameters.

In HCL (HashiCorp Configuration Language), when defining a variable you do not necessarily need to specify anything except its name. You would declare an example variable called test_ip like this:

variable "test_ip" { }

You can then use this value through the code; passing its value in when you run Terraform.

While that will work, this definition has two shortcomings: first, you can not pass a value in at runtime; and second, it can be of any type (bool, string and so on), which may not be suitable for its purpose. To remedy this, you should always specify its default value and type:

variable "test_ip" {
  type    = string
  default = "8.8.8.8"
}

By setting a default value, you ensure that the code referencing the variable remains operational in the event that a more specific value was not provided. When you specify a type, Terraform can validate the new value the variable should be set to, and show an error if it’s non-conforming to the type. An instance of this behavior would be trying to fit a string into a number.

A new feature of Terraform 0.13 is that you can provide a validation routine for variables that can give an error message if the validation fails. Examples of validation would be checking the length of the new value if it’s a string, or looking for at least one match with a RegEx expression in case of structured data.

To add input validation to your variable, define a validation block:

variable "test_ip" {
  type    = string
  default = "8.8.8.8"

  validation {
    condition     = can(regex("d{1,3}.d{1,3}.d{1,3}.d{1,3}", var.test_ip))
    error_message = "The provided value is not a valid IP address."
  }
}

Under validation, you can specify two parameters within the curly braces:

• A condition that accepts a bool it will calculate, which will signify if the validation passes.
• An error_message that specifies the error message in case the validation does not pass.

In this example, you compute the condition by searching for a regex match in the variable value. You pass that to the can function. The can function returns true if the function that’s passed in as a parameter ran without errors, so it’s useful for checking when a function completed successfully or returned results.

The regex function we’re using here accepts a Regular Expression (RegEx), applies it to a given string, and returns the matched substrings. The RegEx matches four pairs of three digit numbers, separated with dots in-between. You can learn more about RegEx by visiting the Introduction to Regular Expressions tutorial.

You now know how to specify a default value for a variable, how to set its type, and how to enable input validation using RegEx expressions.

Conclusion

In this tutorial, you’ve troubleshooted Terraform by enabling debug mode and setting the log verbosity to appropriate levels. You’ve also learned about some of the advanced features of variables, such as declaring validation procedures and setting good defaults. Leaving out default values is a common pitfall that may cause strange issues further along in your project’s development.

For the stability of your project, locking third-party module and provider versions is recommended, since it leaves you with a stable system and an upgrade path when it becomes necessary.

Verification of input values for variables is not confined to matching with regex. For more built-in functions that you can make use of, visit the official docs.

To learn more about Terraform, check out our How To Manage Infrastructure with Terraform series.