Simple Automated Snapshots of Multiple EBS Volumes

I have multiple EBS volumes attached to multiple EC2 instances. I needed a way to take a daily snapshot of all volumes. In addition, I needed the snapshots to rotate, such that only the last 7 days worth of snapshots would be kept.

I chose to create a small EC2 instance specifically for running ec2-automate-backup from a cron job that will backup the volumes of all my production instances.

Setting up the ec2 cli tools

The ec2-automate-backup script depends on the ec2 cli lools. I’m using Ubuntu 12.04, which does not come pre-installed with the ec2-ami-tools and ec2-api-tools. So they will need to be installed. The tools depend on Java, so make sure that’s installed:

sudo apt-get install openjdk-6-jre-headless

Set the JAVA_HOME environment variable.

export JAVA_HOME="/usr/lib/jvm/java-6-openjdk-amd64/jre"

Download and install the ec2 tools

wget http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip
sudo mkdir /usr/local/ec2
sudo unzip ec2-api-tools.zip -d /usr/local/ec2

Set the EC2_HOME environment variable and append it to your path. (Make note of the actual /usr/local/ec2/ install dir. Future versions will have a different directory)

export EC2_HOME=/usr/local/ec2/ec2-api-tools-1.6.13.0
export PATH=$PATH:$EC2_HOME/bin

It’s a good idea to add these to your bash .profile file, so they will be available next time you login. So append the following to the end of .profile

export JAVA_HOME="/usr/lib/jvm/java-6-openjdk-amd64/jre"
export EC2_HOME=/usr/local/ec2/ec2-api-tools-1.6.13.0
export PATH=$PATH:$EC2_HOME/bin

Install ec2-automate-backup

I chose to install this script in /root/bin

#change to root
sudo -s
cd /root

#create bin directory for scripts
mkdir bin
cd bin

wget https://raw.githubusercontent.com/colinbjohnson/aws-missing-tools/master/ec2-automate-backup/ec2-automate-backup.sh

chmod +x ec2-automate-backup.sh

ec2-automate-backup.sh takes a “cron primer” script as an argument. This exports various variables that will be available to cron when the script runs. There’s an example script in the resources directory, so let’s get that too. We’ll edit it to meet our needs.

wget https://raw.githubusercontent.com/colinbjohnson/aws-missing-tools/master/ec2-automate-backup/Resources/cron-primer.sh
chmod +x cron-primer.sh

Edit the cron-primer.sh script to reflect the values of our JAVA_HOME and EC2_HOME environment variables so that they’ll be available to cron. Also add your access key/secret key.

#!/bin/bash -
# EC2_HOME required for EC2 API Tools
export EC2_HOME=/usr/local/ec2/ec2-api-tools-1.6.13.0
# JAVA_HOME required for EC2 API Tools
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk-amd64/jre
# export PATH=/bin is required for cut, date, grep
# export PATH=/opt/aws/bin/ is required for EC2 API Tools
export PATH=/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/ec2/ec2-api-tools-1.6.13.0/bin/
export AWS_ACCESS_KEY=xxxxxxxxxxxxxxx
export AWS_SECRET_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Tag EBS Volumes in your AWS Console

The ec2-automate-backup script can back up any volume that has a certain tag. Thus adding the tag Backup=true to any volume will automatically make it back up. This is nice, since we don’t ever have to touch the cron or any other scripts. We can simply set it and forget it, and manage the Backup=true tag from within the AWS console.

Giving it a whirl

Now that everything is set up, and the ESB volumes are tagged with Backup=true, we can give it a test run with the command:

./ec2-automate-backup.sh -r eu-west-1 -s tag -t 'Backup=true' -k 7 -p -n -c /root/bin/cron-primer.sh

This should take a snapshot of all EBS volues having the tag Backup=true in the eu-west-1 region. Only the newest 7 snapshots will be kept. You can adjust this with the -k parameter.

Add a cron job

If everything works, we can create a cron job that runs daily

0 22 * * * /root/bin/ec2-automate-backup.sh -r eu-west-1 -s tag -t 'Backup=true' -k 7 -p -n -c /root/bin/cron-primer.sh