We will use a custom resource written in Python that will be able to create ACM certificates with DNS validation. The custom resource will also automatically validate this certificate if the validation domain is managed by a Route53 hosted zone. We will also be able to specify an AWS region to create the certificate in, this region is independent of the Cloudformation stack region which for example makes it possible to deploy a certificate in region us-east-1 (to use with cloudfront) while deploying the stack in region eu-west-1. The resource will also provide the certificate arn as an output parameter so it can be used by other resources in the stack. Lastly when you delete the custom resource it will cleanup all validation records and the certificate itself.

Requirements:

• Python3
• Pip
• Bash
• Zip
• An S3 bucket to deploy the custom resource package on
• A hosted zone for the validation record

Implementation:

Let’s get started by downloading all the required code from our GitHub repository.

Step1: Uploading the custom resource package

In this step we are going to prepare the custom resource package and upload it to an S3 bucket.

First we go into the custom resource directory.
cd cloudar-acm-plus-custom-resource

Next we execute a script to install all required dependencies.
sh install_dependencies

Now we are ready to create the package.
sh pack_custom_resource

You will now find the zipfile ‘cloudar-acm-plus-custom-resource.zip’ in ‘cloudar-acm-plus-custom-resource/packed’, upload this zip file to your S3 bucket.

Step2: Creating a Cloudformation template

Now we can create a Cloudformation template in which we use this custom resource to create an ACM certificate.
You can use the template ‘cfn.yaml’ as an example.

First create a Lambda resource as following

Use the name of your bucket for the property ‘S3Bucket’ .

Next we create the custom resource.

We can set the following properties here:

• DomainName: (REQUIRED type:String) The domain name for the acm certificate.
• AdditionalDomains: (OPTIONAL type:List) Additional domains for the acm certificate
• ValidationDomain: (REQUIRED type:string) The domain name for the validation domain of the acm certificate
• HostedZoneId: (REQUIRED type:string) The hosted zone id for the validation domain of the acm certificate
• CertificateRegion: (REQUIRED type:string) The region to deploy the acm certificate in
• IdempotencyToken: (REQUIRED type:string pattern: \w+) The idempotency token for the create call of the acm certificate doc: https://docs.aws.amazon.com/acm/latest/APIReference/API_RequestCertificate.html#ACM-RequestCertificate-request-IdempotencyToken
• CertificateTags: (OPTIONAL type:list) The tags for the acm certificate

In order for the DNS record cleanup and delete certificate functionality to work when you delete the Cloudformation stack it is important to set the following output.

As you can see we can access the arn of the certifcate created by the custom resource with the GetAtt function on the resource.
!GetAtt CreateCertificateCustomResource.certificate_arn

Step3: Deploy the cloudformation

Finally the only thing left to do is deploy the Cloudformation template.
Once the deploy is started Cloudformation will create the Lambda containing the code from step1 and start a custom resource which will create the certificate and validation records. Once the status of the certificate becomes ‘ISSUED’ the custom resource will finish successfully and report the arn of the certificate back to Cloudformation. We can now further use this arn in other resources in the Cloudformation template.
When you delete the Cloudformation stack, the custom resource will cleanup the validation records in the hosted zone and delete the certificate.

CREATE_COMPLETE

The post Validate ACM certificates in Cloudformation appeared first on Cloudar.

Amazon EC2 Systems Manager is a collection of capabilities that helps you automate management tasks such as collecting system inventory, applying operating system patches, automating the creation of Amazon Machine Images (AMIs), and configuring operating systems and applications at scale. It is available at no cost to manage both your EC2 and on-premises resources!

Amazon EC2 Systems Manager relies on the Amazon Simple Systems Management Service (SSM) agent being installed on the guests. The SSM agent is pre-installed on Windows Server 2016 instances or Windows Server 2003-2012 R2 instances created from AMI’s published after November 2016. You need at least SSM agent version 2.0.599.0 installed on the target EC2 instance.

In this article we will focus on using Systems Manager to apply Windows Updates to EC2 instances. Patch Management is always an operational pain point so its welcome that AWS offers a solution.

You start by creating groups of instances by applying a tag called ‘Patch Group’. Then you create a group of patches by forming a patch baseline containing and excluding the patches you require (or use the AWS default patch baseline). At last you create a maintenance window to have your patch baseline attached and applied to a patch group. The actual ‘Patch Now’ run-command is nothing more than an API call, so there’s no obligation to use Maintenance Windows. Personally I’m a fan of Rundeck, so I’ll show you how to have the patches applied to the instances using both methods.

Configure your instances

The guest SSM agent setting inside with Windows OS requires permissions to connect to AWS EC2 Systems Manager. We grant these rights by creating an EC2 Service Role with the policy document ‘AmazonEC2RoleforSSM’ attached. Then you can attach this role to your instances. The instance also needs outbound internet connection to be able to connect to SSM. This can be either through an Internet Gateway or a NAT Gateway (or NAT Instance).

If you have this done right, your instance(s) should pop-up under ‘Managed Instances’ in the EC2 console:

Take note of the SSM Agent Version. As said earlier it must be at least version 2.0.599.0. The Systems Manager Service also requires a “Patch Group”-tag on the EC2 instance. The key for a patch group tag must be Patch Group. Note that the key is case sensitive. The value can be anything you want to specify, but the key must be Patch Group.

If done correctly, your tag will be picked up by SSM. You can confirm this on the ‘Managed Instances’ page:

Patch Baselines

AWS provides a default Patch Baseline called ‘AWS-DefaultPatchBaseline’. It auto-approves all critical and security updates with a ‘critical’ or ‘important’ classification seven days after they have been released by Microsoft. If you’re happy with that you can use this baseline. If you’re not, you can simply create your own according to your requirements: set approval for specific products and patch classifications, exclude a specific KB etc

Once your happy with your baseline, you can hit ‘Create’. Now assign it to one or more Patch Groups (or make it the default baseline and throw away the AWS one). Hit the ‘actions’ menu and chose ‘Modify Patch Groups’

Type the names of the Patch Groups you defined when tagging your instances

Your baseline is now attached to the specified patch groups. You can now start evaluating your instances against the baseline, and update them accordingly.

Patching

Applying the patch baseline to a specific instance or to a patch group is nothing more than executing an AWS SSM run command. You can schedule this run command through AWS SSM ‘Maintenance Windows’, a cron job on a server (like Rundeck) or manual through the AWS Console.

Let’s first check everything manually. In the AWS EC2 console, go to ‘Run Commands’ and create a new Run Command. Select the ‘AWS-ApplyPatchBaseline’ command document and pick an instance run this on. For the ‘operation’, choose ‘Scan’. This will evaluate the instance against the baseline without installing anything yet.

Once the run command finishes, you can go back to the ‘Managed Instances’ page. Highlight the instance(s) on which the run command was executed and click on the ‘Patch’ tab. Here you can see the result of the scan:

To actually install the missing updates, execute the same run command document, but now with the ‘Install’ operation. This will install the missing KBs to the instances and reboot them if needed.

Or execute the following aws cli command to accomplish the same:

aws ssm send-command --targets "Key=tag:Patch Group,Values=<PatchGroupName>" --document-name "AWS-ApplyPatchBaseline" --comment "Install|Check Windows Updates" --parameters Operation="<Install|Scan>"

Maintenance Windows

In stead of manually starting a run command or cron job, we can also use the AWS provided Maintenance Windows feature. Systems Manager Maintenance Windows let you define a schedule for when to perform actions on your instances such as patching the operating system. Each Maintenance Window has a schedule, a duration, a set of registered targets, and a set of registered tasks.

Before actually creating a Maintenance Window, we must configure a Maintenance Window role. We need this so Systems Manager can execute tasks in Maintenance Windows on our behalf. So we go to the IAM page and create a new role. We pick an “EC2 service role” type and make sure to attach the “AmazonSSMMaintenanceWindowRole” policy to it. Once the role is created, we must modify it. Click “edit Trust Relationships”. Add a comma after “ec2.amazonaws.com”, and then add “Service”: “ssm.amazonaws.com” to the existing policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com",
        "Service": "ssm.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Back to SSM now to actually create the Maintenance Window. Give it a useful name and specify your preferred schedule. I’m setting ‘every 30 minutes’ just for demonstration purposes, but in a real setup you would most probably choose something like ‘Every  Sunday’. You can also configure your own Cron expression.

This leaves us now with an empty Maintenance Window: there are no tasks nor targets associated yet.

To assign targets to the Maintenance Window, click on the “Register new targets” button on the “Targets” tab. We dynamically select the targets by using the “Patch Group” tag.

We will now have an ID linked to our “dev” Patch Group. This “Window Target ID” is used in the next step.

From the “tasks” tab of the Maintenance Window, click on “Schedule new task”. Pick the “AWS-ApplyPatchBaseline” document. Under “Registered Targets”, select the correct Window Target ID. For the operation, select “Install”. For the “Role”, select the IAM role with the AmazonSSMMaintenanceWindowRole attached to it (the one we created earlier). Set your preferred concurrency level and register the task by clicking on the blue button. The end result should look like this:

Now we have to wait for the schedule of the Maintenance Window. In this example we specified ‘every 30 minutes’ as a schedule, so the waiting shouldn’t take too long. Under the ‘History’ tab of the Maintenance Window you can follow all actions. The Maintenance Window will simply launch a Run Command, so you could go to that console screen too. If you enabled logging to S3, you could find the output of the Run Command over there. If not, you can view a (truncated) output via the Run Command itself:

Patch Summary for i-07ca5621af38f256d
PatchGroup          : dev
BaselineId          : pb-06101e06cf8506be6
SnapshotId          : 317f2b72-2612-4740-95af-c7b3d8fb6d1e
OwnerInformation    : 
OperationType       : Install
OperationStartTime  : 2017-05-29T11:00:14.0000000Z
OperationEndTime    : 2017-05-29T11:03:18.7164313Z
InstalledCount      : 1
InstalledOtherCount : 6
FailedCount         : 0
MissingCount        : 0
NotApplicableCount  : 3

EC2AMAZ-EA5SH8I - PatchBaselineOperations Installation Results - 2017-05-29T11:03:19.537

KbArticleId Installed   Message
----------- ----------- -----------
KB890830    Yes         Success

If we now go back to the “Managed Instances” page and look at the “Patch” tab of our test instance, we will see it is not missing any updates anymore!

Success! Another  on the Automation checklist!

Rutger

The post Windows servers patching with AWS EC2 Systems Manager appeared first on Cloudar.

Recently we were looking for a way to automate the following actions.

• Creating snapshots
• Starting and stopping instances
• Removing old snapshots

We already had a script and cronjob to take regular snapshots, but we wanted to centralise our automation to improve visibility and logging.

While this is possible using the existing Ansible modules, we needed a way to specify multiple parameters on a per instance basis. Specifically we needed it to be possible to:

• Change the start and stop times from within the AWS Console
• Allow users without access to (or knowledge of) Ansible to prevent the automatic shutdown of a machine

We quickly decided to use tags to specify these parameters. Because there’s a limit of ten tags per instance, and we wanted to store other information (like Environment and Project) too, it wasn’t possible to store each parameter in a separate tag.

We solved this by writing our own Ansible modules. Each module reads the same tag, where we store a JSON dictionary with all the needed variables. This way we keep room for other tags, while allowing everyone with the right to create tags to change things if they need to.

The source code is available on GitHub. You can install it by adding the path to your Ansible Library. Complete usage instructions and examples are included in the repository.

The post Flexible Instance and Snapshot management with Ansible appeared first on Cloudar.

Recently we were looking for a way to have our EC2 EBS volumes snapshotted on a daily basis. Although AWS makes it easy to take snapshots of your EBS volumes, this requires some manual intervention and thus prone to human error.
As we love to automate as much as possible (we firmly believe in the ‘set & forget’ principle), we were looking for a way to script the AWS snapshotting mechanism. A quick Google search showed us that there already many solutions available that can handle this. However, one of these really drew our attention: The ec2-automate-backup script by Collin Johnson. The script has it all:

• snapshot one specific volume or all volumes
• Choose which volumes to backup
• Backup volumes for as specific region
• Snapshot retention: delete snapshots older than x days
• …

To handle automated snapshots on a daily basis of all our EBS volumes, we’ve setup a t1.micro EC2 instance (running CentOS 6.5) on which we created a cron job that runs daily, inventarises all volumes and snapshots only those volumes that have a Backup TAG value set to TRUE. Snapshots are being retained for 7 days by default. If the script detects snapshots older than 7 days, they will be deleted.

This is how we’ve set this up:

Prerequisites

Java

The Amazon EC2 CLI tools require Java. If you don’t have Java 1.7 or later installed, download and install Java as shown below:

yum install java-1.7.0-openjdk

Run the file command recursively to find the binary:

file $(which java)

→ example output: /usr/bin/java: symbolic link to ‘/etc/alternatives/java’

The ‘/usr/bin/java location’ is actually a link to ‘/etc/alternatives/java’, so you need to run the file command on that location to see whether that is the real binary:

file /etc/alternatives/java

→ example output: /etc/alternatives/java: symbolic link to `/usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java’

This returns a new location, which is the actual binary. Verify this by running the file command on this location:

file /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java

→ example output: /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, stripped
This location is the actual binary. The Java home directory is where bin/java lives; in this example, the Java home directory is ‘/usr/lib/jvm/jre-1.7.0-openjdk.x86_64’.

You will now have to set the JAVA_HOME variable to the full path of the Java home directory:

export JAVA_HOME=/usr/lib/jvm/jre-1.7.0-openjdk.x86_64

You can verify your JAVA_HOME setting using this command:

$JAVA_HOME/bin/java -version

That’s it. We can now continue installing the AWS EC2 CLI tools.

Amazon EC2 CLI Tools

Download EC2 API tools

#Go to temp directory
cd /tmp

#Download the latest version of the EC2 CLI tools
wget http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip

#Unzip the downloaded file
unzip ec2-api-tools.zip

Install EC2 API tools

#Create directory to install the EC2 CLI tools
mkdir /usr/local/ec2/apitools

#Move the downloaded tools to custom directory (replace version with your own version if other)
mv ec2-api-tools-1.7.1.1/* /usr/local/ec2/apitools

Set variables

#Set the EC2_HOME environment variable:
export EC2_HOME=/usr/local/ec2/apitools

#Update the PATH environment variable
export PATH=$PATH:$EC2_HOME/bin

Add variables to startup script, so they are widely available (Optional)

#Go to profile.d directory
cd etc/profile.d

#Create a new bash script
vi aws.sh

You can now add the variables into the script (we use vi as default editor):

export JAVA_HOME=/usr/lib/jvm/jre-1.7.0-openjdk.x86_64
export EC2_HOME=/usr/local/ec2/apitools
export PATH=$PATH:$EC2_HOME/bin

Save and close the file

Create IAM Backup User

It’s a common best practice to create a separate account with limited access rights for these kind of purpose.

• Go to the IAM Console
• Create a user “backup-ebs-user” (or a name of your own choice)

→ Follow the AWS instructions for adding a new IAM User
→ Take a note of the user and security credentials for future usage

• Assign policy to the backup user, so only required access rights are assigned to the user

→ Follow AWS instructions for an overview of the IAM user polices

• Below is a sample policy that contains the IAM permissions required to run ec2-automate-backup.sh

{
	"Statement": [
		{
			"Action": [
				"ec2:DescribeVolumes",
				"ec2:CreateSnapshot",
				"ec2:DescribeSnapshots",
				"ec2:DeleteSnapshot",
				"ec2:CreateTags",
				"ec2:DescribeTags"
			],
			"Effect": "Allow",
			"Resource": [
				"*"
			]
		}
	]
}

Install the ec2-automate-backup script

#Go to EC2 directory
cd /usr/local/ec2

#Create script directory
Mkdir scripts

#Go to script directory
Cd scripts

# Download the script
wget https://raw.githubusercontent.com/colinbjohnson/aws-missing-tools/master/ec2-automate-backup/ec2-automate-backup.sh

#Make the downloaded file executable by everyone
chmod +x ec2-automate-backup.sh

The ec2-automate-backup script allows you to specify a source file for environmental configuration. This is interesting for running the script as a cronjob. An example cron primer file is located at https://github.com/colinbjohnson/aws-missing-tools/tree/master/ec2-automate-backup/Resources.
Let’s download the file and modify it according to our specific needs:

#Download the cron-primer.sh example file
wget https://raw.githubusercontent.com/colinbjohnson/aws-missing-tools/master/ec2-automate-backup/Resources/cron-primer.sh

#Make the downloaded file executable by everyone
chmod +x cron-primer.sh

Now edit the cron-primer.sh script so custom variables are available to cron
→ (don’t forget to add your Access/Secret key)
Our edited cron-primer.sh file:

#!/bin/bash -
# EC2_HOME required for EC2 API Tools
export EC2_HOME=/usr/local/ec2/apitools
# JAVA_HOME required for EC2 API Tools
export JAVA_HOME=/usr/lib/jvm/jre-1.7.0-openjdk.x86_64
# export PATH=/bin is required for cut, date, grep
# export PATH=/opt/aws/bin/ is required for EC2 API Tools
export PATH=/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/ec2/apitools/bin/
export AWS_ACCESS_KEY=xxxxxxxxxxxxxxxxxxxxxxxxx
export AWS_SECRET_KEY=xxxxxxxxxxxxxxxxxxxxxxxxx

Add custom Backup TAG to your EBS Volumes

Because we only want to backup the volumes that have a Backup Tag value set to ‘true’, we need to add these tags to our EBS volumes. This has the advantage of leaving the scripts untouched and it enables us to manage the volumes we want snapshot through the AWS console.
Just add the Backup TAG and set its value to ‘true’ as shown in the example below:

Create Cron job

In the example below, we created a cron job that runs daily at 03:00AM.

0 03 * * * /usr/local/ec2/scripts/ec2-automate-backup.sh -r eu-west-1 -s tag -t 'Backup=true' -k 7 -p -n -c /usr/local/ec2/scripts/cron-primer.sh

This cron job creates a snapshot of all EBS volumes which reside in the ‘eu-west-1’ region and that have the Backup Tag value set to ‘true’. Snapshots are being retained for 7 days.

We hope this blogpost can be of some assistance in helping you to automate some manual tasks. Feel free to comment or modify the script for your own purposes.

References we used

→ https://github.com/colinbjohnson/aws-missing-tools/tree/master/ec2-automate-backup

→ http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/set-up-ec2-cli-linux.html

The post Automating snapshots of EC2 EBS volumes appeared first on Cloudar.