Test Terraform with Kitchen and AWSpec

19Aug

Test Terraform with Kitchen and AWSpec

Test Terraform with Kitchen

In an ideal world, you write code and then test it, period. Doesn’t matter what kind of code, you just test it, even if it’s Terraform we’re talking about. One might say that Terraform is not a real language and I can’t argue with that but its code, manifest, whatever you want to call it, generates resources that will affect, one way or another, your stack. That’s why I never stopped making researches trying to find the right way to test it. I was told that terraform plan would do the job although I have always claimed this is a very bold statement as it can catch some errors – mainly syntax or interpolation ones - but ignore so many others. I researched and researched until I found my answer; guess what, I’m going to test my code using test-kitchen, again! But it wasn’t enough, I wanted to unit test it too. AWSpec, RSpec based is what I needed. So, in case you’re still wondering, I am going to talk about how to test Terraform code using Test Kitchen and awspec.

Goals

Let’s define our MVP’s before starting. With this post we will:

  • Build a VPC Terraform module to provision VPC components and subnets (public and private)
  • Create a test manifest that includes the module we just created
  • Provision our test VPC using test kitchen
  • Write some unit tests using awspec and verify our stack by running kitchen verify

NB: test-kitchen will provision the stack to your AWS account, therefore, you’ll be charged for the time you keep it alive.

Requirements:

• Terraform ~0.11.x
• Ruby >= 2.2.x
• Bundler gem (gem install bundler)

VPC Module

As a good practice, I tend to write reusable modules parameterizing everything I can so that:
• my code is more DRY; and
• they can be published as open source projects.

As defined in our MVP’s, we want to create a module that can be used to create an AWS VPC and its subnets so first let’s create our working directory:

$ mkdir -p terraform-modules/networking/vpc
$ cd terraform-modules/networking/vpc

Now we can start to write our module in our main.tf file:

resource "aws_vpc" "vpc" {
  cidr_block       = "${var.vpc_cidr_block}"
  instance_tenancy = "${var.vpc_instance_tenancy}"

  tags = "${
        merge(
            map("Name", format("%s-vpc", var.environment)),
            map("Environment", format("%s", var.environment)),
            map("Service", format("%s", var.service)),
            var.vpc_tags, var.tags
        )
    }"
}

# Public Subnets
resource "aws_subnet" "public-subnet" {
  count             = "${length(data.aws_availability_zones.all.names)}"
  vpc_id            = "${aws_vpc.vpc.id}"
  availability_zone = "${element(data.aws_availability_zones.all.names, count.index)}"
  cidr_block        = "${cidrsubnet(aws_vpc.vpc.cidr_block, 8, count.index + 101)}"

  tags = "${
        merge(
            map("Name", format("%s-public-subnet-%d", var.environment, count.index)),
            map("Environment", format("%s", var.environment)),
            map("Service", format("%s", var.service)),
            var.tags
        )
    }"
}

# Private subnets
resource "aws_subnet" "private-subnet" {
  count             = "${length(data.aws_availability_zones.all.names)}"
  vpc_id            = "${aws_vpc.vpc.id}"
  availability_zone = "${element(data.aws_availability_zones.all.names, count.index)}"
  cidr_block        = "${cidrsubnet(aws_vpc.vpc.cidr_block, 8, count.index + 1)}"

  tags = "${
        merge(
            map("Name", format("%s-private-subnet-%d", var.environment, count.index)),
            map("Environment", format("%s", var.environment)),
            map("Service", format("%s", var.service)),
            var.tags
        )
    }"
}

The purpose of this post is not to talk about this terraform code and what it means but, in case you are confused by some bits of it, e.g. data.aws_availability_zones.all.names, here’s what my data.tf file looks like:

data "aws_availability_zones" "all" {}

It lists all the availability zones for a given AWS region.

The next step is to create a variables.tf file in which we’ll list the variables used by this module adding a description and, eventually, some default values:

# …
variable "tags" {
  description = "A map of tags to add to all resources"
  default     = {}
}
# ...

Install test-kitchen and awspec

Before proceeding, let’s create a Gemfile, in the root of the project, to install our testing tools:

# frozen_string_literal: true

ruby '2.4.2'

source 'https://rubygems.org/' do
  gem 'aws-sdk', '~> 3.0.1'
  gem 'awspec', '~> 1.4.0'
  gem 'kitchen-terraform', '~> 3.1'
  gem 'kitchen-verifier-awspec', '~> 0.1.1'
  gem 'rhcl', '~> 0.1.0' # ruby hcl parser
end

To install those packages, type the following command:

$ bundle install --path vendor/bundle

Your gems will be scoped to the project only and they won’t be accessible from any other part of your Filesystem. If, by any chance, you want to install them globally, type this command:

$ sudo bundle install

Test the module with test-kitchen

Now that our module is done, we’re ready to test it using test-kitchen. Firstly, we have to write a test manifest that includes the VPC module and this file will live in a test/test_fixture folder:

provider "aws" {
  region = "us-east-1"
}

data "aws_availability_zones" "all" {}

module "vpc" {
  source               = "../../"
  vpc_cidr_block       = "10.0.0.0/16"
  vpc_instance_tenancy = "default"
  environment          = "test"
  service              = "kitchen-vpc"

  tags = {
    test        = "this-is-a-test-tag"
    Description = "Managed by Terraform"
  }

  vpc_tags = {
    vpc-tag = "thisIsAVPCTag"
  }
}

And now it’s time to create our .kitchen.yml file:

---
driver:
  name: "terraform"
  root_module_directory: "test/test_fixture"

provisioner:
  name: "terraform"

platforms:
  - name: "aws"

verifier:
  name: "awspec"

suites:
  - name: "default"

I bet that if you’ve used test-kitchen before, this file looks familiar to you, right? Just notice the driver name: that’s telling test-kitchen to use the kitchen-terraform plugin we previously installed.

Now, time to see if our module actually works:

$ bundle exec kitchen converge

Boom, test-kitchen just created a test environment for you, ready to be unit-tested :).

Unit Tests with awspec

Let’s create the folder we’ll create the unit test file in to:

$ mkdir -p test/integration/default

And finally, let’s create a file called test_vpc.rb:

# frozen_string_literal: true

require 'awspec'
require 'aws-sdk'
require 'rhcl'

# Parse and load our terraform manifest into example_main
example_main = Rhcl.parse(File.open('test/test_fixture/main.tf'))
vpc_name = example_main['module']['vpc']['environment'] + "-vpc"
environment_tag = example_main['module']['vpc']['environment']
service_tag = example_main['module']['vpc']['service']
description_tag = "Managed by Terraform"
# Load the terraform state file and convert it into a Ruby hash
state_file = 'terraform.tfstate.d/kitchen-terraform-default-aws/terraform.tfstate'
tf_state = JSON.parse(File.open(state_file).read)
region = tf_state['modules'][0]['outputs']['region']['value']
ENV['AWS_REGION'] = region

# Get AZ Names using the AWS SDK
ec2 = Aws::EC2::Client.new(region: region)
azs = ec2.describe_availability_zones
zone_names = azs.to_h[:availability_zones].map { |az| az[:zone_name] }

# Test the VPC resource
describe vpc(vpc_name.to_s) do
  it { should exist }
  it { should be_available }
  it { should have_tag('Name').value(vpc_name.to_s) }
  it { should have_tag('Environment').value(environment_tag.to_s) }
  it { should have_tag('Service').value(service_tag.to_s) }
  it { should have_tag('Description').value(description_tag.to_s) }
end

# Loop through the AZ’s and test the private and public subnets
zone_names.each_with_index do |az, index|
  describe subnet("#{environment_tag.to_s}-public-subnet-#{index}") do
    it { should exist }
    it { should be_available }
    it { should belong_to_vpc(vpc_name.to_s) }
    it { should have_tag('Name').value("#{environment_tag}-public-subnet-#{index}") }
    it { should have_tag('Environment').value(environment_tag.to_s) }
  end

  describe subnet("#{environment_tag.to_s}-private-subnet-#{index}") do
    it { should exist }
    it { should be_available }
    it { should belong_to_vpc(vpc_name.to_s) }
    it { should have_tag('Name').value("#{environment_tag}-private-subnet-#{index}") }
    it { should have_tag('Environment').value(environment_tag.to_s) }
  end
end

Just a few notes about this file:

  • The use of the rhcl allows us to get information dynamically without the need of hardcode values in our test file;
  • Unfortunately, some information can’t be retrieved from the manifest, so we have to parse the tfstate file;
  • You can test literally anything with awspec, be creative and think of what you’d like to unit-test;
  • The awspec resource types list can be found here.

It’s now time to verify our infrastructure but first we have to tell test-kitchen where to find the spec file. Open the .kitchen.yaml file and modify the suites block so it looks like this:

suites:
  - name: "default"
    verifier:
      name: "awspec"
      patterns:
      - "test/integration/default/test_vpc.rb"

Finally, type in your terminal:

$ bundle exec kitchen verify

Congrats, you’ve tested your terraform module \o/. Now, since you’re paying for those resources on AWS, maybe it’s best to get rid of them:

$ bundle exec kitchen destroy

Conclusion

Now you know how to test your Terraform code, think about how many things you could do with it. For instance, you could think to create automated tests using your favourite CI tool and promote the module, or main manifest, through dev, staging and production. Or, here’s another idea, you could also implement some integration tests too. Since RSpec is nothing but ruby, you can write your own code to check some of your stack functionalities, like making requests to a Load Balancer and check the response, query a database, etc. Now go out there and spread the word: test all the things!

Spread the love