Foreword

Before backing up resources in OpenStack, it's important to understand the difference between two primary methods:

1. Snapshots: These are delta copies of the root volumes attached to the VM. Snapshots are not full copies and rely on the original volume for restoration, so they are mostly useful for accident recovery. They're managed through the Cinder service.
2. Backups: These are point-in-time copies of the volumes attached to VMs. Backups preserve the data within a volume and can be used to restore the volume to the exact state it was in when backed up. They're ideal for data recovery and are managed through the Cinder service.

Choose the appropriate method based on your recovery needs: snapshots for accident prevention and recovery or backups for full data security and recovery.

VM snapshots

To take a snapshot of a VM for later recovery or replication, one must take a snapshot of the root and other attached volumes as well as store the image configuration.  The list of VMs/servers than can be used for the snapshot can be found with:

openstack server list

First one needs to find out the volumes attached to the instance, with

openstack server show -c volumes_attached <server>

with <server>  the name or id of the VM to backup. This will output a list of attached volumes like 

Alternatively, one can list all volumes present in the cloud with

openstack volume list

which will also inform about which volumes are mounted on which VMs. For instance

Then, to create a snapshot of one of the volumes, do

openstack volume snapshot create [--force] --volume <volume> <snapshot_name>

with:

• --force :  create the snapshot even if the volume is attached
• –volume <volume> : the name or id of the volume to be backed up
• <snapshot_name> : the name of the new volume snapshot

To see the list of all snapshots in your cloud, or the snapshots from a specific volume, run

openstack volume snapshot list [--volume <volume>]

and to get the details of a specific snapshot

openstack volume snapshot show <snapshot_name>

It is recommended to detach the volume from the server before making a snapshot, to avoid potential crashes. This can be done with the command

openstack server remove volume <server> <volume>

and it can be reattached after the snapshot with

openstack server add volume <server> <volume>

Note that in order to make a snapshot without detaching the volume the --force  flag must be provided. The snapshot or backup of an attached volume is crash consistent.

One can also store the VM configuration for later recovery, simply by exporting the details of the instance to a file, for example with

openstack server show <sever> > instance-config.txt

VM Backups

The procedure to create a backup of a VM is almost identical to that of a snapshot. First identify the volumes attached to the VM as before, then back up each of the volumes with

openstack volume backup create [--incremental] [--force] [--name <backup_name>] <volume>

where:

• --incremental: create an incremental backup (requires a previous full backup)
• --force: create a backup even if the volume is attached
• --name <backup_name>: the name of the new volume backup
• <volume>: the name or id of the volume to be backed up

To list all available backups in your cloud, you can use the following command:

openstack volume backup list

Additionally, you can filter the results with additional options to list backups for a specific volume:

 openstack volume backup list --volume <volume>

It is recommended to detach the volume from the server before backing it up, to avoid potential crashes. This can be done as explained above.  One can force the backup of attached volumes with the --force  option. The snapshot or backup of an attached volume is crash consistent.

As before, it is also possible to store the instance configuration data for later recovery.

Common use cases

 These is a list of common configurations of VMs in the cloud and the recommened procedure to backing them up

Single VM

To backup a single VM, it is recommended to create a snapshot of the root volume of the VM, as shown above. A full backup is also recommended when the original volume is not guaranteed to be preserved.

Single VM and attached volume(s)

To backup a single VM and it's attached volumes, one must create a snapshot or backup of the root volume of the VM as well as other attached volumes separately. 

Single VM and multiple networks and/or security groups

The network configuration and security groups of VMs are not recorded when performing an snapshot or backup of the root volume of the VM. If this information needs to be backed up in order to reconfigure the VM after restoration, we recommend storing these values in a file. These can be done as before when storing the full configuration of a VM, e.g. in YAML format, with

openstack server show -f yaml -c addresses -c security_groups <instance> > <file>

From these stored details, the network and security group configuration can be restored. Visit EWC - How to restore backups from VMs to find out more about how to restore VMs.

Single VM and SFS

Shared File Services (SFS) cannot be backed up as they are distributed storage solutions that do not live in the same cloud. The recommended way to backup a VM with a SFS is to unmount the SFS before performing the snapshot or backup of the VM, in order to avoid possible corruption, and remounting it afterwards. This can be easily done, with admin privileges, with

sudo umount <sfs_mount_point>
...
<backup volumes, etc>
...
sudo mount <sfs_url> <sfs_mount_point>

Multiple VMs, volumes, etc

In order to backup multiple VMs, with any configuration, one can repeat the steps laid out above multiple times in succession. Attention must be when backing up VMs with multiple volumes so as not to lose information. For the sake of safety, it is recommended to detach all the volumes and stop all the instances that are to be backed up.

An alternative to repeating all the steps here multiple times is to use the automation script below, where one can set up a configuration file detailing all the VMs, volumes, etc, to be backed up, and run the script a single time.

Scheduling backups

It is often quite useful to schedule a backup for a specific time in the future, as well as to repeat that scheduled backup regularly, to ensure the preservation of any changes. In order to prepare a scheduled backup, with or without repetition, one can set up a cron  job that will run in the background and perform the necessary operations. A full tutorial on how to use cron is beyond the scope of this page, more details can be found elsewhere, e.g. https://crontab.guru/. Here only a few examples will be shown.

First, it is worth reminding that it it advisable that the creation of snapshots and backups is done in a different machine that the one backing up. Consequently, the cron  job should also be run in a different machine. To run a cron  job, ensure that cron is installed in the machine, e.g.

 sudo apt update && sudo apt install cron 

in Ubuntu distributions, and start the cron services with

sudo systemctl enable cron

Once cron is running, to configure a cron job, one must edit the crontab  file with

crontab -e

and add there the command to be executed and the frequency. For example, to create an snapshot of the root volume of a VM every Saturday at 12pm, the crontab entry will look like

0 12 * * Sat openstack volume snapshot create --force --volume <volume> <snapshot_name>

.Note that this job will necessarily fail, since the cron service does not use the environment variables with the OpenStack credentials that are needed to use the OpenStack CLI. Hence, a better solution is to write a shell script that will load the appropriate environment variables and run any operations. Thus, the same example with the correct OS credentials will look like

0 12 * * Sat run_backup_job.sh 2>&1 | logger -t backups

which will run the script run_backup_job.sh  and log all the output. The script run_backup_job.sh is then

#! /usr/bin/env bash

for var in $(env | sed -n 's/^\(OS.*\)=.*/\1/p'); do unset "$var"; done
export OS_AUTH_URL=https://keystone.cloudferro.com:5000/v3
export OS_INTERFACE=public
export OS_IDENTITY_API_VERSION=3
export OS_USERNAME="<username>"
export OS_REGION_NAME="WAW3-1"
export OS_PROJECT_ID="<project_id>
export OS_PROJECT_NAME="<project_name>"
export OS_PROJECT_DOMAIN_ID="<domain_id>"
export OS_PASSWORD="<password>"
export OS_CLIENT_ID=openstack
export OS_CLIENT_SECRET="<client_secret>"
export OS_PROTOCOL=openid
export OS_IDENTITY_PROVIDER=eumetsat_provider
export OS_AUTH_TYPE=v3oidcpassword
export OS_DISCOVERY_ENDPOINT="https://identity.cloudferro.com/auth/realms/Eumetsat-elasticity/.well-known/openid-configuration"

echo "Backing up VM"
sleep 5
openstack volume snapshot create --force --volume 49f0613f-ef9b-4d91-b963-e958579801ed TestVM_snapshot

To run the backups only once, without repetition, it is possible to set up a conditional statement on the script, such as

...
<set up env>
...

COMMAND=run_backup_job.sh
DONE="${COMMAND}.alreadyrun"
   
if [[ -f $DONE ]]; then
  exit 1
fi
touch "$DONE"

...
<run some commands>
...

Alternatively, the same result can be obtained using the at command instead of cron , by running

./run_backup_job.sh | at <some_future_time>

Lastly, when scheduling backups, it is recommended to set up a retention limit, so that no more than a certain number of backups are kept at a given time. One can incorporate this check, and the corresponding deletion of older backups, in the cron  script, with

...
<do some backup>
...

retention_count=5
backups=(`openstack volume snapshot list --volume <root_volume_id> --sort-column updated_at --sort-ascending -c ID -f value`)
nbackups=${#backups[@]}
echo "Found $nbackups backups of VM"
if [[ $nbackups -gt $retention_count ]]
then
    let "todelete=$nbackups-$retention_count"
    echo "Backup number over the retention count, will delete the oldest $todelete backups"
    for ((i=0;i<todelete;i++)); do
        echo "Deleting snapshot ${backups[$i]}"
        openstack volume snapshot delete ${backups[$i]}
    done;
else
	echo "Backup number equal or under retention count, doing nothing"
fi

Note: some of the syntax above is specific to bash . Add SHELL=/usr/bin/env bash to your crontab  to ensure cron runs using bash , or change the above script for its equivalent in sh .

Backup automation

The multi-step procedure for creating backups, including the scheduling and repetition of backups, can been performed in a single step using the script found in the repository

EWC User Tools

under the items/openstack_backups directory.  A simple configuration YAML file detailing the VMs to back up, as well as the start and frequency of the scheduling, can be used to run the script. Instructions on how to run the script can be found in the README file in the repository itself.