Using Ansible to provision AWS autoscaling instances
If you are running your infrastructure on EC2 in AWS, you are probably using AWS’s autoscaling service to manage your instances. If you are not, you should be. Even if you don’t plan to have your instances scale up and down based on traffic patterns, you’ll want to use autoscaling for the simple case of keeping a certain number of instances running if one of them stops responding or is decommissioned by AWS – without requiring manual intervention by an engineer to spin up and configure a new server.
The tricky bit is making sure that the new instances that are created are configured properly for their purpose. For Cassandra, for example, this might mean pointing the configuration at the seed nodes for your cluster. For a web server this may mean configuring which database to point to and credentials for connecting. Either way it is rare that your server will need no configuration at launch time to operate properly (even though you should attempt to embed as much configuration as possible into your AMIs). AWS’s solution for this is user-data, a script (or cloud-init file) that is executed on first boot.
The simplest thing you can do is provide a bash script, e.g.
#!/bin/bash aws ec2 describe-instances \ --filters "Name=tag:Database,Values=true" \ --query 'Reservations[*].Instances[*].PrivateDnsName' \ --output text \ > /etc/databases systemctl start my-webservice
This is a contrived example, but I hope it is illustrative.
At Braintree, as our user-data became more complex (things like configuring firewall rules, setting up certificates and configuring services like exhibitor), we made the decision to migrate from bash scripts to Ansible configuration to better organize it and take advantage of jinja2 templating.
If you are familiar with Ansible, you can probably guess that the difficulty with using Ansible as user-data is that Ansible configuration typically consists of a playbook and all of its supporting assets (templates, static files, roles, etc.) which doesn’t neatly fit into a single file (AWS also limits user-data to 16 KB) so it may seem like your only option is to store the configuration elsewhere (like S3) and retrieve it in your user-data or bake it into your image (slow turn around time for changes), but below I’ll describe another option that ended up working for us.
Compress and base64 encode your Ansible configuration! It may seem like
a simple idea, but it didn’t immediately occur to us so I thought I would share
it here. I also think this technique could be used for other standalone
configuration management tools like
chef-solo or even just modularized
We use Terraform to manage our infrastructure, so we have this compression and
base64 encoding happen as a preprocessing step when planning (using a wrapper
terraform plan) that is used by our launch configuration
resources, but you could fit it into your workflow however works best.
Example script you can use to automate creation of the user-data script:
#! /bin/sh ansible_configuration=$1 shift cat <<-EOF #! /bin/bash set -o errexit mkdir -p /tmp/ansible echo '$(tar c -C "$ansible_configuration" . | gzip -n | base64 -w 0)' | base64 -d | tar xz -C /tmp/ansible cd /tmp/ansible EOF echo -n '/usr/local/bin/ansible-playbook playbook.yml --connection=local -i localhost, -e target=localhost'
Note that this script assumes that you have a variable called
target that you
use for your
hosts declaration in your playbook.
Example of generated user-data (base64 encoded directory shortened for brevity):
#! /bin/bash set -o errexit mkdir -p /tmp/ansible echo 'H4sIAAAAAAAAA+w9aXPbRpb+[...]+CV+qgxijv8ggqkPOrhZngzP99N/Tv0z+6bZbGw3u9QXa2v1ovOfw0nfP/yPxjlBAKBQCAQCAQCgUAgEAgEAoHg+eEnXr53cwCgAAA=' | base64 -d | tar xz -C /tmp/ansible cd /tmp/ansible /usr/local/bin/ansible-playbook playbook.yml --connection=local -i localhost, -e target=localhost
This is what you would use for the user-data in your launch configuration
whereby, on launch, the encoded Ansible configuration is decoded, written