08/09/2014 Bart Van Hecke

Automating snapshots of EC2 EBS volumes

Note: We stopped using this script, and started using Ansible to create snapshots. You can read more about that here.

Recently we were looking for a way to have our EC2 EBS volumes snapshotted on a daily basis. Although AWS makes it easy to take snapshots of your EBS volumes, this requires some manual intervention and thus prone to human error.
As we love to automate as much as possible (we firmly believe in the ‘set & forget’ principle), we were looking for a way to script the AWS snapshotting mechanism. A quick Google search showed us that there already many solutions available that can handle this. However, one of these really drew our attention: The ec2-automate-backup script by Collin Johnson. The script has it all:

  • snapshot one specific volume or all volumes
  • Choose which volumes to backup
  • Backup volumes for as specific region
  • Snapshot retention: delete snapshots older than x days

To handle automated snapshots on a daily basis of all our EBS volumes, we’ve setup a t1.micro EC2 instance (running CentOS 6.5) on which we created a cron job that runs daily, inventarises all volumes and snapshots only those volumes that have a Backup TAG value set to TRUE. Snapshots are being retained for 7 days by default. If the script detects snapshots older than 7 days, they will be deleted.

This is how we’ve set this up:

Prerequisites

Java

The Amazon EC2 CLI tools require Java. If you don’t have Java 1.7 or later installed, download and install Java as shown below:

Run the file command recursively to find the binary:

→ example output: /usr/bin/java: symbolic link to ‘/etc/alternatives/java’

The ‘/usr/bin/java location’ is actually a link to ‘/etc/alternatives/java’, so you need to run the file command on that location to see whether that is the real binary:

→ example output: /etc/alternatives/java: symbolic link to `/usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java’

This returns a new location, which is the actual binary. Verify this by running the file command on this location:

→ example output: /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, stripped
This location is the actual binary. The Java home directory is where bin/java lives; in this example, the Java home directory is ‘/usr/lib/jvm/jre-1.7.0-openjdk.x86_64’.

You will now have to set the JAVA_HOME variable to the full path of the Java home directory:

You can verify your JAVA_HOME setting using this command:

That’s it. We can now continue installing the AWS EC2 CLI tools.

 

Amazon EC2 CLI Tools

Download EC2 API tools

 

Install EC2 API tools

 

Set variables

 

Add variables to startup script, so they are widely available (Optional)

You can now add the variables into the script (we use vi as default editor):

Save and close the file

Create IAM Backup User

It’s a common best practice to create a separate account with limited access rights for these kind of purpose.

  • Go to the IAM Console
  • Create a user “backup-ebs-user” (or a name of your own choice)

→ Follow the AWS instructions for adding a new IAM User
→ Take a note of the user and security credentials for future usage

  • Assign policy to the backup user, so only required access rights are assigned to the user

→ Follow AWS instructions for an overview of the IAM user polices

  • Below is a sample policy that contains the IAM permissions required to run ec2-automate-backup.sh

 

Install the ec2-automate-backup script

The ec2-automate-backup script allows you to specify a source file for environmental configuration. This is interesting for running the script as a cronjob. An example cron primer file is located at https://github.com/colinbjohnson/aws-missing-tools/tree/master/ec2-automate-backup/Resources.
Let’s download the file and modify it according to our specific needs:

Now edit the cron-primer.sh script so custom variables are available to cron
→ (don’t forget to add your Access/Secret key)
Our edited cron-primer.sh file:

 

Add custom Backup TAG to your EBS Volumes

Because we only want to backup the volumes that have a Backup Tag value set to ‘true’, we need to add these tags to our EBS volumes. This has the advantage of leaving the scripts untouched and it enables us to manage the volumes we want snapshot through the AWS console.
Just add the Backup TAG and set its value to ‘true’ as shown in the example below:

Backup Tags

 

Create Cron job

In the example below, we created a cron job that runs daily at 03:00AM.

This cron job creates a snapshot of all EBS volumes which reside in the ‘eu-west-1’ region and that have the Backup Tag value set to ‘true’. Snapshots are being retained for 7 days.

We hope this blogpost can be of some assistance in helping you to automate some manual tasks. Feel free to comment or modify the script for your own purposes.

References we used

https://github.com/colinbjohnson/aws-missing-tools/tree/master/ec2-automate-backup

http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/set-up-ec2-cli-linux.html

Tagged: , , , , , , , ,

Comments (10)

  1. Thanks for your contribution! It works perfect!
    I just had to change the tag -t ‘Backup=true’ by tag -t ‘Backup,Values=true’ . Otherwise describe-volumes didn’t detect my volumes with tag Backup on true.

  2. Wes

    Hi Roger,

    I finally got it working, however, I would like the name of all my instances to be included, in the snapshot, as volume-id is less desirable. how would I modify the code to do this?

  3. trent

    No unmounting or freezing of the filesystem as AWS recommends? The script unfortunately isn’t that useful to run centrally since it can’t go freeze all the filesystems while the snapshot runs.

  4. Ben Bridts

    Hi Trent,

    The snapshots are indeed not guaranteed to be consistent. They will only contain the data that’s written to disk. If you require consistency when creating snapshots (eg. for your database servers), have a look at https://github.com/alestic/ec2-consistent-snapshot . The setup is more complex though.

    Recently we stopped using this script, and started using Ansible to create snapshots. You can read more about that here: http://www.cloudar.be/awsblog/instance-and-snapshot-management-with-ansible/ . The Ansible modules do not support consistent snapshots at the moment, but it’s definitely something we want to implement in the future.

Leave a Reply

Your email address will not be published. Required fields are marked *