Foreword

OpenStack recognises two types of resources on their clouds:

1. Instances, which correspond to the servers or VMs. These can be image-backed, if the VM was booted from an image, or volume-backed, if it was booted from a volume. Instances are managed via the OS Glance service.
2. Volumes, which correspond to storage disks which can be attached to VMs. Root disk are a special case as VMs are booted from them. Volumes are managed by the OS Cinder service.

Before backing up resources in OpenStack, it's important to understand the difference between two primary methods:

1. Snapshots: These are delta copies of the VMs or volumes. Snapshots are not full copies and rely on the original resource for restoration, so they are mostly useful for accident recovery.
2. Backups: These are point-in-time copies of the VMs or volumes. Backups preserve the data within a resource and can be used to restore it to the exact state it was in when backed up. They're ideal for data recovery.

Choose the appropriate method based on your recovery needs: snapshots for accident prevention and recovery or backups for full data security and recovery.

Backing up VMs

Instance Snapshots

There are two methods of taking snapshots of instances, depending on whether they are image-backed or volume-backed. 

Image-backed instances

To take a snapshot of a  image-backed instance for later recovery or replication, one must create an image based on the current VM.  The list of VMs/servers than can be used for the snapshot can be found with:

openstack server list

And, to create the image snapshot of a running VM with name or id <server> use the command

openstack server image create --name <image_name> <server>

where:

• --name <image_name>: the name for the new image snapshot
• <server>: the name or id of your server / VM to be backed up.

In order to see the list of images, including the recently created snapshot, one can run

openstack image list

and the specific details of the created image can be found

openstack image show <image_name>

It is recommended, though not mandatory, to shut off the server before making a snapshot of it, to make sure all the data has been flushed to disk. This can be done with the command

openstack server stop <server>

and after the snapshot it can be restarted with

openstack server start <server>

Volume-backed instances

To take a snapshot of a volume-backed instance for later recovery or replication, one must take a snapshot of the root and other attached volumes as well as store the image configuration.  The list of VMs/servers than can be used for the snapshot can be found with:

openstack server list

First one needs to find out the volumes attached to the instance, with

openstack server show -c volumes_attached <server>

with <server>  the name or id of the VM to backup. This will output a list of attached volumes like 

Alternatively, one can list all volumes present in the cloud with

openstack volume list

which will also inform about which volumes are mounted on which VMs. For instance

Then, to create a snapshot follow the instructions on how to take a snapshot of a volume below.

One can also store the VM configuration for later recovery, simply by exporting the details of the instance to a file, for example with

openstack server show <sever> > instance-config.txt

Instance Backups

Image-backed instances

One can perform a full backup of an image-backed instance with

openstack server backup create <name_or_id> --name <backup_name>

with <name_or_id> the name or id of the instance to back up and <backup_name> the name of the backup to create.

As before, the list of backups can be browsed with

openstack image list

and it is recommended to stop the server before the backup.

Volume-backed instances

Performing a full backup of a volume-backed VM requires backing up the root volume. As before, identify the root volume of the VM, and then perform a full backup, as explained in the instructions below.

Backing up Volumes

Volume Snapshots

To create a snapshot of a volume, root or otherwise, do

openstack volume snapshot create [--force] --volume <volume> <snapshot_name>

with:

• --force :  create the snapshot even if the volume is attached
• –volume <volume> : the name or id of the volume to be backed up
• <snapshot_name> : the name of the new volume snapshot

To see the list of all snapshots in your cloud, or the snapshots from a specific volume, run

openstack volume snapshot list [--volume <volume>]

and to get the details of a specific snapshot

openstack volume snapshot show <snapshot_name>

It is recommended to detach the volume from the server before making a snapshot, to avoid potential crashes. This can be done with the command

openstack server remove volume <server> <volume>

and it can be reattached after the snapshot with

openstack server add volume <server> <volume>

Note that in order to make a snapshot without detaching the volume the --force  flag must be provided. The snapshot or backup of an attached volume is crash consistent.

Volume Backups

The procedure to create a backup of a volume is almost identical to that of a snapshot. First identify the volumes in the cloud as before, then back up each of the volumes with

openstack volume backup create [--incremental] [--force] [--name <backup_name>] <volume>

where:

• --incremental: create an incremental backup (requires a previous full backup)
• --force: create a backup even if the volume is attached
• --name <backup_name>: the name of the new volume backup
• <volume>: the name or id of the volume to be backed up

To list all available backups in your cloud, you can use the following command:

openstack volume backup list

Additionally, you can filter the results with additional options to list backups for a specific volume:

 openstack volume backup list --volume <volume>

It is recommended to detach the volume from the server before backing it up, to avoid potential crashes. This can be done as explained above.  One can force the backup of attached volumes with the --force  option. The snapshot or backup of an attached volume is crash consistent.

Common use cases

 These is a list of common configurations of VMs in the cloud and the recommened procedure to backing them up

Single VM

To backup a single VM, whether image-backed or volume-backed, it is recommended to create a snapshot of the instance as shown above. A full backup is also recommended for volume-backed instances when the original volume is not guaranteed to be preserved.

Single VM and attached volume(s)

To backup a single VM and it's attached volumes, one must create a snapshot or backup of the VM first, including the root disk if the instance is volume-backed, and then perform a snapshot or backup of all other attached volumes separately. 

Single VM and multiple networks and/or security groups

The network configuration and security groups of VMs are not recorded when performing an snapshot or backup of the VM. If this information needs to be backed up in order to reconfigure the VM after restoration, we recommend storing these values in a file. These can be done as before when storing the full configuration of a VM, e.g. in YAML format, with

openstack server show -f yaml -c addresses -c security_groups <instance> > <file>

From these stored details, the network and security group configuration can be restored. Visit EWC - How to restore backups from VMs to find out more about how to restore VMs.

Single VM and SFS

Shared File Services (SFS) cannot be backed up as they are distributed storage solutions that do not live in the same cloud. The recommended way to backup a VM with a SFS is to unmount the SFS before performing the snapshot or backup of the VM, in order to avoid possible corruption, and remounting it afterwards. This can be easily done, with admin privileges, with

sudo umount <sfs_mount_point>
...
<backup volumes, etc>
...
sudo mount <sfs_url> <sfs_mount_point>

Multiple VMs, volumes, etc

In order to backup multiple VMs, with any configuration, one can repeat the steps laid out above multiple times in succession. Attention must be when backing up VMs with multiple volumes so as not to lose information. For the sake of safety, it is recommended to detach all the volumes and stop all the instances that are to be backed up.

An alternative to repeating all the steps here multiple times is to use the automation script below, where one can set up a configuration file detailing all the VMs, volumes, etc, to be backed up, and run the script a single time.

Scheduling backups

It is often quite useful to schedule a backup for a specific time in the future, as well as to repeat that scheduled backup regularly, to ensure the preservation of any changes. In order to prepare a scheduled backup, with or without repetition, one can set up a cron  job that will run in the background and perform the necessary operations. A full tutorial on how to use cron is beyond the scope of this page, more details can be found elsewhere, e.g. https://crontab.guru/. Here only a few examples will be shown.

First, it is worth reminding that it it advisable that the creation of snapshots and backups is done in a different machine that the one backing up. Consequently, the cron  job should also be run in a different machine. To run a cron  job, ensure that cron is installed in the machine, e.g.

 sudo apt update && sudo apt install cron 

in Ubuntu distributions, and start the cron services with

sudo systemctl enable cron

Once cron is running, to configure a cron job, one must edit the crontab  file with

crontab -e

and add there the command to be executed and the frequency. For example, to create an snapshot of the root volume of a VM every Saturday at 12pm, the crontab entry will look like

0 12 * * Sat openstack volume snapshot create --force --volume <volume> <snapshot_name>

.Note that this job will necessarily fail, since the cron service does not use the environment variables with the OpenStack credentials that are needed to use the OpenStack CLI. Hence, a better solution is to write a shell script that will load the appropriate environment variables and run any operations. Thus, the same example with the correct OS credentials will look like

0 12 * * Sat run_backup_job.sh 2>&1 | logger -t backups

which will run the script run_backup_job.sh  and log all the output. The script run_backup_job.sh is then

#! /usr/bin/env bash

export OS_AUTH_TYPE=v3applicationcredential
export OS_AUTH_URL=https://keystone.api.r1.cloud.eumetsat.int
export OS_IDENTITY_API_VERSION=3
export OS_REGION_NAME="ECIS-R1"
export OS_INTERFACE=public
export OS_USERNAME="<username>" 
export OS_USER_DOMAIN_NAME="<domain_name>"
export OS_PROJECT_DOMAIN_ID="<domain_id>"
export OS_APPLICATION_CREDENTIAL_ID="<app_credential_id>"
export OS_APPLICATION_CREDENTIAL_SECRET="<app_credential_secret>"

echo "Backing up VM"
sleep 5
openstack volume snapshot create --force --volume 49f0613f-ef9b-4d91-b963-e958579801ed TestVM_snapshot

To run the backups only once, without repetition, it is possible to set up a conditional statement on the script, such as

...
<set up env>
...

COMMAND=run_backup_job.sh
DONE="${COMMAND}.alreadyrun"
   
if [[ -f $DONE ]]; then
  exit 1
fi
touch "$DONE"

...
<run some commands>
...

Alternatively, the same result can be obtained using the at command instead of cron , by running

./run_backup_job.sh | at <some_future_time>

Lastly, when scheduling backups, it is recommended to set up a retention limit, so that no more than a certain number of backups are kept at a given time. One can incorporate this check, and the corresponding deletion of older backups, in the cron  script, with

...
<do some backup>
...

retention_count=5
backups=(`openstack volume snapshot list --volume <root_volume_id> --sort-column updated_at --sort-ascending -c ID -f value`)
nbackups=${#backups[@]}
echo "Found $nbackups backups of VM"
if [[ $nbackups -gt $retention_count ]]
then
    let "todelete=$nbackups-$retention_count"
    echo "Backup number over the retention count, will delete the oldest $todelete backups"
    for ((i=0;i<todelete;i++)); do
        echo "Deleting snapshot ${backups[$i]}"
        openstack volume snapshot delete ${backups[$i]}
    done;
else
	echo "Backup number equal or under retention count, doing nothing"
fi

Note: some of the syntax above is specific to bash . Add SHELL=/usr/bin/env bash to your crontab  to ensure cron runs using bash , or change the above script for its equivalent in sh .

Backup automation

The multi-step procedure for creating backups, including the scheduling and repetition of backups, can been performed in a single step using the script found in the repository

EWC User Tools

under the items/openstack_backups directory.  A simple configuration YAML file detailing the VMs to back up, as well as the start and frequency of the scheduling, can be used to run the script. Instructions on how to run the script can be found in the README file in the repository itself.