As testing new cookbooks directly in opsworks takes a long time, it makes a lot of sense to test them in vagrant first. There is some documentation available on how to do this, but some of it is out of date, broken or unclear.
I’ve tried to cobble something together that’s quick to set up and works good enough (aka: not 100% identical to opsworks, but close enough)
To mimic an opsworks environment in vagrant, we need several tools:
- vagrant
- test kitchen
- berkshelf
- chef
Below we describe one way to create and test a new cookbook. There is another way (by using the opsworks provisioner) which we’ll use later. For now, the standard chef_zero provisioner works well enough.
The next setup allows us to test one single cookbook. If you need to test several cookbooks at the same time, things have to change slightly. More about this later.
Creating a project
mkdir [project name]
cd !$
mkdir cookbooks
berks cookbook [cookbook name] ## create the skeleton for a new cookbook
Modify the .kitchen.yml file:
---
driver:
name: vagrant
require_chef_omnibus: 11.10.4 ## opsworks still uses this version of chef
provisioner:
name: chef_zero ## chef_solo is deprecated
environments_path: ./environments ## this folder contains the json file that mimics the opsworks environment data
platforms:
- name: centos-6.7 ## most similar to the current amazon linux
suites:
- name: testsuite
run_list:
- recipe[cookbook name::recipe]
attributes: ## standard way to provide attributes to cookbooks, but it's better to use the environments functionality, that way we can just copy and paste the json data from opsworks to vagrant
tester:
test: "hello"
provisioner: ## Opsworks style attributes
client_rb:
environment: test ## the name of the actual file containing our json data
To obtain the opsworks attributes, create an opsworks stack + host, log in to that host, and run:
sudo opsworks-agent-cli get_json
This command will provide you the full list of attributes. It’s better to keep only the attributes we need and transfer those over to our environments file. However, the cool thing about opsworks is all hosts can access all data about the stack from these attributes, so it’s worth checking them out in detail to see what’s available.
The example below contains the bare minumum to get things working.
{
"default_attributes": {
"opsworks" : {
"stack" : {
"name" : "MyStack",
"id" : "42dfd151-6766-4f1c-9940-ba79e5220b58"
}
}
},
## custom attributes have to be added here and can be accessed with node['custom attribute']
## in opsworks you can just specify, in the custom json field, { 'custom attribute': 'value' }
"chef_type" : "environment",
"json_class" : "Chef::Environment"
}
Next, if our module has dependencies to other modules, we need to set up our berksfile.
source "https://supermarket.chef.io"
metadata
cookbook 'supermarket_cookbook'
You also need to add the dependencies in the metadata.rb file
name 'cookbook_name'
maintainer 'YOUR_COMPANY_NAME'
maintainer_email 'YOUR_EMAIL'
license 'All rights reserved'
description 'Installs/Configures my cookbook'
long_description IO.read(File.join(File.dirname(__FILE__), 'README.md'))
version '0.1.0'
depends 'supermarket_cookbook'
Test
We didn’t actually specify any real tests in this example (check out the test kitchen manual on how to set this up), but we can provision our test machine with the following command:
kitchen converge
Once the machine has converged (or not), we can log into the instance to do some manual debugging with:
kitchen login
Just as a sidenote: if you want to debug on opsworks, all interesting files are under /opt/aws/opsworks. Even better: you can modifie your cookbooks directly under /opt/aws/opsworks/current/site-cookbooks and run the opsworks-agent-cli with the correct options to rerun chef with your modified cookbooks.
Testing multiple cookbooks at once
There is an important difference between how berksfiles are interpreted by opsworks. You need a different Berksfile and metadata per environment
As a reminder, this is our directory layout:
.
└── project
├── Berksfile # testkitchen
├── .kitchen.yml # testkitchen
└── cookbooks
├── Berksfile # opsworks
├── cookbook-a
│ ├── Berksfile.in # testkitchen
│ └── metadata.rb # opsworks + testkitchen
└── cookbook-b
├── Berksfile.in # testkitchen
└── metadata.rb # opsworks + testkitchen
One special remark: Opsworks needs a git repo with only the Berksfile and the two cookbook directories, so take care about how you commit to git!
The main Berksfile for testkitchen is the one under /project/. This calls the Berksfile.in in every cookbook.
I found this code somewhere on the internet, but can’t find the link anymore, so my apologies about not giving credit where credit is due!
source "https://supermarket.chef.io"
# Note the absence of the metadata line.
def dependencies(path)
berks = "#{path}/Berksfile.in"
instance_eval(File.read(berks)) if File.exists?(berks)
end
Dir.glob('./cookbooks/*').each do |path|
dependencies path
cookbook File.basename(path), :path => path
end
The sub Berksfiles only contain the needed cookbooks for that specific cookbook. The metadata.rb file is automatically sourced by the cookbook
cookbook "dependency-a"
cookbook "dependency-b"
Opsworks needs the following file. Again no metadata is referenced as opsworks will find the necessary files automatically.
source "https://supermarket.chef.io"
cookbook "dependency-a"
cookbook "dependency-b"
This is the ‘normal’ Berksfile for cookbook-a. You still need this if you want to test it independently!
source "https://supermarket.chef.io"
metadata
cookbook "dependency-a"
References
http://pixelcog.com/blog/2014/virtualizing-aws-opsworks-with-vagrant/ (this uses the opsworks provisioner, didn’t get it working yet)
http://enriquecordero.com/programming/opsworks-chef-workflow/
http://docs.aws.amazon.com/opsworks/latest/userguide/opsworks-opsworks-mock.html (outdated)
https://www.youtube.com/watch?v=0sPuAb6nB2o (very interesting test kitchen / berkshelf tutorial)