Test kitchen opsworks workflow

26/09/2015
Posted in AWS Blog
26/09/2015 Wannes Van Causbroeck

As testing new cookbooks directly in opsworks takes a long time, it makes a lot of sense to test them in vagrant first. There is some documentation available on how to do this, but some of it is out of date, broken or unclear.

I’ve tried to cobble something together that’s quick to set up and works good enough (aka: not 100% identical to opsworks, but close enough)

To mimic an opsworks environment in vagrant, we need several tools:

  • vagrant
  • test kitchen
  • berkshelf
  • chef

Below we describe one way to create and test a new cookbook. There is another way (by using the opsworks provisioner) which we’ll use later. For now, the standard chef_zero provisioner works well enough.
The next setup allows us to test one single cookbook. If you need to test several cookbooks at the same time, things have to change slightly. More about this later.

Creating a project


mkdir [project name]
cd !$
mkdir cookbooks 
berks cookbook [cookbook name]  ## create the skeleton for a new cookbook

 

Modify the .kitchen.yml file:


---
driver:
  name: vagrant
  require_chef_omnibus: 11.10.4  ## opsworks still uses this version of chef
provisioner:
  name: chef_zero  ## chef_solo is deprecated
  environments_path: ./environments  ## this folder contains the json file that mimics the opsworks environment data
platforms:
  - name: centos-6.7  ## most similar to the current amazon linux
suites:
  - name: testsuite
    run_list:
      - recipe[cookbook name::recipe]
    attributes:  ## standard way to provide attributes to cookbooks, but it's better to use the environments functionality, that way we can just copy and paste the json data from opsworks to vagrant
    tester:
      test: "hello"
   provisioner:  ## Opsworks style attributes
     client_rb:
       environment: test ## the name of the actual file containing our json data

 

To obtain the opsworks attributes, create an opsworks stack + host, log in to that host, and run:


sudo opsworks-agent-cli get_json

 

This command will provide you the full list of attributes. It’s better to keep only the attributes we need and transfer those over to our environments file. However, the cool thing about opsworks is all hosts can access all data about the stack from these attributes, so it’s worth checking them out in detail to see what’s available.
The example below contains the bare minumum to get things working.


{
  "default_attributes": {
    "opsworks" : {
      "stack" : {
        "name" : "MyStack",
        "id" : "42dfd151-6766-4f1c-9940-ba79e5220b58"
      }
    }
  },
  ## custom attributes have to be added here and can be accessed with node['custom attribute']
  ## in opsworks you can just specify, in the custom json field, { 'custom attribute': 'value' }
  "chef_type" : "environment",
  "json_class" : "Chef::Environment"
}

 

Next, if our module has dependencies to other modules, we need to set up our berksfile.


source "https://supermarket.chef.io"
 
metadata

cookbook 'supermarket_cookbook'

 

You also need to add the dependencies in the metadata.rb file


name             'cookbook_name'
maintainer       'YOUR_COMPANY_NAME'
maintainer_email 'YOUR_EMAIL'
license          'All rights reserved'
description      'Installs/Configures my cookbook'
long_description IO.read(File.join(File.dirname(__FILE__), 'README.md'))
version          '0.1.0'
 
depends 'supermarket_cookbook'

Test

We didn’t actually specify any real tests in this example (check out the test kitchen manual on how to set this up), but we can provision our test machine with the following command:


kitchen converge

 

Once the machine has converged (or not), we can log into the instance to do some manual debugging with:


kitchen login

 

Just as a sidenote: if you want to debug on opsworks, all interesting files are under /opt/aws/opsworks. Even better: you can modifie your cookbooks directly under /opt/aws/opsworks/current/site-cookbooks and run the opsworks-agent-cli with the correct options to rerun chef with your modified cookbooks.

Testing multiple cookbooks at once

There is an important difference between how berksfiles are interpreted by opsworks. You need a different Berksfile and metadata per environment
As a reminder, this is our directory layout:


.
└── project
    ├── Berksfile   # testkitchen
    ├── .kitchen.yml   #  testkitchen
    └── cookbooks
        ├── Berksfile   # opsworks
        ├── cookbook-a
        │   ├── Berksfile.in   # testkitchen
        │   └── metadata.rb   # opsworks + testkitchen
        └── cookbook-b
            ├── Berksfile.in   # testkitchen
            └── metadata.rb   # opsworks + testkitchen

 

One special remark: Opsworks needs a git repo with only the Berksfile and the two cookbook directories, so take care about how you commit to git!
The main Berksfile for testkitchen is the one under /project/. This calls the Berksfile.in in every cookbook.
I found this code somewhere on the internet, but can’t find the link anymore, so my apologies about not giving credit where credit is due!


source "https://supermarket.chef.io"
 
# Note the absence of the metadata line.
 
def dependencies(path)
  berks = "#{path}/Berksfile.in"
  instance_eval(File.read(berks)) if File.exists?(berks)
end
Dir.glob('./cookbooks/*').each do |path|
  dependencies path
  cookbook File.basename(path), :path => path
end

 

The sub Berksfiles only contain the needed cookbooks for that specific cookbook. The metadata.rb file is automatically sourced by the cookbook


cookbook "dependency-a"

cookbook "dependency-b"

 

Opsworks needs the following file. Again no metadata is referenced as opsworks will find the necessary files automatically.


source "https://supermarket.chef.io"
 
cookbook "dependency-a"
cookbook "dependency-b"

 

This is the ‘normal’ Berksfile for cookbook-a. You still need this if you want to test it independently!


source "https://supermarket.chef.io"
 
metadata
 
cookbook "dependency-a"

 

References

http://pixelcog.com/blog/2014/virtualizing-aws-opsworks-with-vagrant/ (this uses the opsworks provisioner, didn’t get it working yet)
http://enriquecordero.com/programming/opsworks-chef-workflow/
http://docs.aws.amazon.com/opsworks/latest/userguide/opsworks-opsworks-mock.html (outdated)
https://www.youtube.com/watch?v=0sPuAb6nB2o (very interesting test kitchen / berkshelf tutorial)

  • SHARE

Leave a Reply

Your email address will not be published. Required fields are marked *

LET'S WORK
TOGETHER

Need a hand? Or a high five?
Feel free to visit our offices and come say hi
… or just drop us a message

We are ready when you are

Cloudar – Operations

Veldkant 7
2550 Kontich (Antwerp)
Belgium

info @ cloudar.be

+32 3 450 67 18

Cloudar – HQ

Veldkant 33A
2550 Kontich (Antwerp)
Belgium

VAT BE0564 763 890

    This contact form is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

    contact
    • SHARE