This is a text-only version of the following page on https://raymii.org:
---
Title : OpenStack: Quick and automatic instance snapshot backup and restore (and before an apt upgrade) with nova backup
Author : Remy van Elst
Date : 20-12-2016
URL : https://raymii.org/s/tutorials/OpenStack_Quick_and_automatic_instance_snapshot_backups.html
Format : Markdown/HTML
---
![openStack Logo][1]
This is a guide that shows you how to create OpenStack instance snapshots
automatically, quick and easy. This allows you to create a full backup of the
entire instance. This guide has a script that makes creating snapshots from an
OpenStack VM automatic via cron. The script uses the `nova backup` function,
therefore it also has retention and rotation of the backups. It also features an
option to create a snapshot before every apt action, upgrade/install/remove.
This way, you can easily restore from the snapshot when something goes wrong
after an upgrade. Snapshots are very usefull to restore the entire instance to
an earlier state. Do note that this is not the same as a file based backup, you
can't select a few files to restore, it's all or nothing.
OpenStack Compute (Nova) has a feature that allows you to create a snapshot of a
(running) instance, namely `nova image-create`. This uses the underlying
hypervisor to create an instant snapshot of the virtual machine. In the case of
KVM/qemu this is a qemu-snapshot. This image is then stored in the Glance image
store, allowing you to create a new VM based on the image, thus restoring the
backup or creating a clone. If you use floating IP's, it's a matter of assigning
the floating IP to the new VM and you're all set.
<p class="ad"> <b>Recently I removed all Google Ads from this site due to their invasive tracking, as well as Google Analytics. Please, if you found this content useful, consider a small donation using any of the options below:</b><br><br> <a href="https://leafnode.nl">I'm developing an open source monitoring app called Leaf Node Monitoring, for windows, linux & android. Go check it out!</a><br><br> <a href="https://github.com/sponsors/RaymiiOrg/">Consider sponsoring me on Github. It means the world to me if you show your appreciation and you'll help pay the server costs.</a><br><br> <a href="https://www.digitalocean.com/?refcode=7435ae6b8212">You can also sponsor me by getting a Digital Ocean VPS. With this referral link you'll get $100 credit for 60 days. </a><br><br> </p>
![][3]
> Snapshot before an apt install/upgrade
Snapshots are a fast and easy way to clone or restore an instance to an earlier
point. Let's say you're doing an upgrade of your application and it goes
horribly wrong. Rollback fails and it's the middle of the night. Do you want to
go and restore all the files and databases from the backup of 18 hours earlier,
manually? Or do you want to revert the entire thing to a full snapshot from
before the upgrade? If I could choose, I would know.
Do note that I still recommend you create regular backups. They allow for a more
fine-grained way to restore files/folders/databases and have things like
compression and incremental/differential support, thus backing up only the
things that have changed since the last backup. This snapshot method is just a
copy of the enitre disk, taking up more space.
### Nova Backup
The compute part of OpenStack, nova, has multiple options to create images
(snapshots) of an instance. One is with the `nova image-create` command. This
creates an instant snapshot and uploads that to Glance, the image storage
component.
The second option is `nova backup`. This is almost the same as `image-create`,
with the addition of rotation of the backup snapshots. There are two extra
parameters, `backup_type` and `rotation`. The [documentation][4] is a bit sparse
on this, but the [api][5] and the [source code][6] provide more detail.
The backups are normal glance images. Based on the `backup_type` a set number of
images is saved. Once there are more images of that type then specified in
`rotation`, the oldest is deleted. If we create a backup every day with
`backup_type` set to `daily` and `rotation` set to `7`, on the 8'th day, the
oldest image will be automatically removed (on creation of the new backup).
There is no scheduling in OpenStack, you need to create these backups yourself,
from `cron` for example. The script allows you to specify a `backup_type` and a
`rotation`. The example sets up the following scheme:
* Daily snapshot, `backup_type` set to `daily`, `rotation` set to `7`.
* Weekly snapshot, `backup_type` set to `weekly`, `rotation` set to `4`.
* Monthly snapshot, `backup_type` set to `montly`, `rotation` set to `12`.
* Yearly snapshot, `backup_type` set to `yearly`, `rotation` set to `2`.
The snapshots after an apt upgrade, if configured, have `backup_type` set to
`apt` and `rotation` set to `7`. If no options are provided, the `backup_type`
is set to `manual` and the rotation to `7`. If just a `backup_type` is provided,
the `rotation` defaults to `7`.
We do need to schedule this in `cron` on the VM itself. You could also adapt the
script to run on a management VM to backup all the VM's in an account.
Below is a screenshot of an example daily snapshot setup. We see 7 snapshots
created with rotation set to 7:
![][7]
The 8'th snapshot is created:
![][8]
Only after the 8'th snapshot has succeeded, the 1st snapshot is removed
automatically:
![][9]
If for whatever reason the new snapshot fails, the old one isn't deleted.
In OpenStack the operator can configure where the Glance images are stored. In
the case of my prefered OpenStack provider ([CloudVPS][10], non-affiliate link)
this is their object store (swift). I happen to know that that object store is
replicated over three data centers, making the snapshot storage extremely
redundant.
The script works on both CentOS and Ubuntu, the apt-upgrade snapshot part only
on Ubuntu. Yum has no such pre-hooks as far as I know of. On CentOS you're
better of just executing it manually before a `yum update`.
### OpenStack command line tools and dependencies
The script runs on the server you want to snapshot itself. It's required to have
the command line tools installed and to have a credentials file. First install
the dependencies:
# Ubuntu
apt-get install dmidecode wget python-pip
# CentOS
yum install dmidecode wget python-pip
Recent Ubuntu releases have the OpenStack command line tools packaged:
apt-get install python-keystoneclient python-glanceclient python-novaclient
On older versions and CentOS you can use `pip`, the Python Pypi package tool:
pip install python-keystoneclient python-glanceclient python-novaclient
You also need a credentials file (`computerc`). Below you'll find an example
file, fill it in with the credentials your OpenStack provider sent you.
nano /root/.openstack_snapshotrc
export OS_AUTH_URL="https://identity.stack.cloudvps.com/v2.0"
export OS_TENANT_NAME="PROJECT_UUID"
export OS_TENANT_ID="PROJECT_UUID"
export OS_USERNAME="USERNAME"
export OS_PASSWORD="PASSWORD"
Test if the file is correct by sourcing it in the shell and issueing a command:
source /root/.openstack_snapshotrc
nova credentials
Example output:
+------------------+-------------------------+
| User Credentials | Value |
+------------------+-------------------------+
| id | f3[...]11 |
| name | image-test-1 |
| roles | [{"name": "compute"}] |
| roles_links | [] |
| username | image-test-1 |
+------------------+-------------------------+
### Install the script
The script is on my Github and can be downloaded with `wget`:
wget -O "/usr/local/bin/create-snapshot.sh" "https://github.com/RaymiiOrg/openstack-nova-snapshot/blob/master/create_snapshot.sh"
Set the executable permission on the file:
chmod +x /usr/local/bin/create-snapshot.sh
The script is simple and accepts two parameters, namely the `backup_type` and
`rotation`. They are not required, by default it will give the `manual` backup
type and a rotation of `7`. For your daily backup with 7 snapshots saved you can
use the following:
/usr/local/bin/create-snapshot.sh daily 7
For your apt snapshots you should give another type:
/usr/local/bin/create-snapshot.sh apt 7
The auto-removal is based on the `backup_type` field, so make sure you give the
correct one. Otherwise you might just delete other backups.
If the credentials file is set up you should now be able to create a snapshot:
/usr/local/bin/create-snapshot.sh
Output:
INFO: Start OpenStack snapshot creation.
SUCCESS: Backup image created and pending upload.
The state of the server will change:
![][11]
Do note that while the image is being uploaded you cannot create a new snapshot
or do other management actions like reboot or shut down, unless you do that via
the VM itself (console).
The image will be queued first and then start uploading:
![][12]
### Backup snapshot schedule
Now that the snapshots are working we can use cron to schedule them As discussed
earlier, we have a schedule for a daily, weekly, monthly and yearly backup. You
could just limit it to a daily schedule if you want to save space. Or for an
important machine, create a snapshot every two hours during office hours, change
it to suit your requirement.
Place the following in a file in `/etc/cron.d/`
# Daily snapshot
1 1 * * * root /bin/bash /usr/local/bin/create-snapshot.sh daily 7
# Weekly snapshot
1 1 * * 0 root /bin/bash /usr/local/bin/create-snapshot.sh weekly 4
# Montlhy snapshot
1 1 1 * * root /bin/bash /usr/local/bin/create-snapshot.sh monthly 12
# Yearly snapshots
1 1 1 1 * root /bin/bash /usr/local/bin/create-snapshot.sh yearly 2
This enables the schedule as described above. If you for example want to have a
snapshot every two hours between office hours you can use the following cron
syntax:
# Snapshot past every 2nd hour from 6 through 18
1 6-18/2 * * * root /bin/bash /usr/local/bin/create-snapshot.sh officehours 12
### Apt hook
`apt`, the package manager, has a hook named `pre-invoke` (and `post-invoke`).
This hook is called before before (or after) `apt` calls `dpkg`. [via][13].
We can use this hook to create the snapshot before an `apt` action, say an `apt-
get upgrade`, `apt-get install` or `apt-get remove`. Create the following file:
nano /etc/apt/apt.conf.d/00openstacksnapshot
Insert the following:
DPkg::Pre-Invoke {"/bin/bash /usr/local/bin/create-snapshot.sh apt 5";};
This will create a snapshot before the actual action is done, allowing you to
revert to a snapshot after a (failed) upgrade. For example when the kernel is
not working or other issues.
![][3]
> Snapshot before an apt install/upgrade
Try it out by installing a package. You should now see the output as in the
above screenshot added to the `apt` command output.
### Restore
The most important thing of a backup is of course that you can restore it when
needed. We can do this via the command line client with the `nova boot` command.
This creates a new instance based on the snapshot. Do note that you need to
source the credentials file in your shell:
source .computerc
nova boot --image "SNAPSHOT_NAME" --poll --flavor "FLAVOR_NAME" --availability-zone NL1 --nic net-id=NETWORK_ID --key "SSH_KEY" "VM_NAME"
The syntax is specific to CloudVPS but you get the gist. If you have a floating
IP you can deassign it from the old server and assign it to the new one and
you're all set.
If you don't have a floating IP or other reasons to keep the same VM, you can
use the `nova rebuild` command. This will reinstall the VM with the snapshot
image, _OVERWRITING ALL DATA ON THE INSTANCE_!:
nova rebuild --poll "INSTANCE_UUID" "SNAPSHOT_IMAGE_UUID"
Afterwards your snapshot image should be deployed on the instance.
Remember to test this often. I test my backups and restores at least once a
month.
[1]: https://raymii.org/s/inc/img/openstack.jpg
[2]: https://www.digitalocean.com/?refcode=7435ae6b8212
[3]: https://raymii.org/s/inc/img/glance_apt.png
[4]: http://docs.openstack.org/cli-reference/nova.html
[5]: http://developer.openstack.org/api-ref/compute/?expanded=create-server-back-up-createbackup-action-detail
[6]: https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/create_backup.py
[7]: https://raymii.org/s/inc/img/glance-0.png
[8]: https://raymii.org/s/inc/img/glance-1.png
[9]: https://raymii.org/s/inc/img/glance-2.png
[10]: http://cloudvps.com
[11]: https://raymii.org/s/inc/img/glance_nova_image.png
[12]: https://raymii.org/s/inc/img/glance_upload.png
[13]: https://wiki.debian.org/AptConf
---
License:
All the text on this website is free as in freedom unless stated otherwise.
This means you can use it in any way you want, you can copy it, change it
the way you like and republish it, as long as you release the (modified)
content under the same license to give others the same freedoms you've got
and place my name and a link to this site with the article as source.
This site uses Google Analytics for statistics and Google Adwords for
advertisements. You are tracked and Google knows everything about you.
Use an adblocker like ublock-origin if you don't want it.
All the code on this website is licensed under the GNU GPL v3 license
unless already licensed under a license which does not allows this form
of licensing or if another license is stated on that page / in that software:
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Just to be clear, the information on this website is for meant for educational
purposes and you use it at your own risk. I do not take responsibility if you
screw something up. Use common sense, do not 'rm -rf /' as root for example.
If you have any questions then do not hesitate to contact me.
See https://raymii.org/s/static/About.html for details.