Pre-requisite: EWC - OpenStack Command-Line client

Foreword

Before backing up resources in OpenStack, it's important to understand the difference between three primary methods:

Intance Snapshots (Glance): These capture the entire virtual machine, including the operating system, applications, and configuration. Instance snapshots are useful for cloning VMs, creating templates, or disaster recovery scenarios. They're managed through the Glance service.
Volume Snapshots (Cinder): These are delta copies of the block storage volumes. Volume snapshots are not full copies and rely on the original volume for restoration, so they are mostly useful for accident recovery. They're managed through the Cinder service.
Volume Backups (Cinder): These are point-in-time copies of block storage volumes. Volume backups preserve the data within a volume and can be used to restore the volume to the exact state it was in when backed up. They're ideal for data recovery and are managed through the Cinder service.

Choose the appropriate method based on your recovery needs: instance snapshots for full system recovery or replication, volume snapshots for accident prevention and recovery or volume backups for full data security and recovery.

Before starting make sure you are in the machine where you configured openstack client and you can run openstack commands as described in EWC - OpenStack Command-Line client

When creating backups or snapshots of VMs and volumes, it is highly recommended that the following command are run outside from said VMs and volumes. Doing so from within may cause corrupted backups or snapshots.

Backup VMs and volumes

Instance Snapshots

To take a snapshot of a instance for later recovery or replication, one must create an image based on the current VM. The list of VMs/servers than can be used for the snapshot can be found with:

openstack server list

And, to create the image snapshot of a running VM with name or id <server> use the command

openstack server image create --name <image_name> <server>

where:

--name <image_name>: the name for the new image snapshot
<server>: the name or id of your server / VM to be backed up.

In order to see the list of images, including the recently created snapshot, one can run

openstack image list

and the specific details of the created image can be found

openstack image show <image_name>

It is recommended, though not mandatory, to shut off the server before making a snapshot of it, to make sure all the data has been flushed to disk. This can be done with the command

openstack server stop <server>

and after the snapshot it can be restarted with

openstack server start <server>

Volume Snapshots

To create a snapshot of a volume

openstack volume snapshot create [--force] --volume <volume> <snapshot_name>

with:

--force : create the snapshot even if the volume is attached
–volume <volume> : the name or id of the volume to be backed up
<snapshot_name> : the name of the new volume snapshot

To see the list of all snapshots in your cloud, or the snapshots from a specific volume, run

openstack volume snapshot list [--volume <volume>]

and to get the details of a specific snapshot

openstack volume snapshot show <snapshot_name>

It is recommended to detach the volume from the server before making a snapshot, to avoid potential crashes. This can be done with the command

openstack server remove volume <server> <volume>

and it can be reattached after the snapshot with

openstack server add volume <server> <volume>

Note that in order to make a snapshot without detaching the volume the --force flag must be provided. The snapshot or backup of an attached volume is crash consistent.

Volume Backups

To create a backup of a volume run:

openstack volume backup create [--incremental] [--force] [--name <backup_name>] <volume>

Where:

--incremental: create an incremental backup (requires a previous full backup)
--force: create a backup even if the volume is attached
--name <backup_name>: the name of the new volume backup
<volume>: the name or id of the volume to be backed up

To list all available backups in your cloud, you can use the following command:

openstack volume backup list

Additionally, you can filter the results with additional options to list backups for a specific volume:

 openstack volume backup list --volume <volume>

It is recommended to detach the volume from the server before backing it up, to avoid potential crashes. This can be done as explained above. One can force the backup of attached volumes with the --force option. The snapshot or backup of an attached volume is crash consistent.

Common use cases

These is a list of common configurations of VMs and volumes in the cloud and the recommened procedure to backing them up

Single VM

To backup a single VM, it is recommended to create an image snapshot of the instance, as shown above. It is also possible to back up the root disk of the VM as a volume, using either a snapshot or full backup, but not recommended.

Single VM and attached volume(s)

To backup a single VM and it's attached volumes, one must create a snapshot or backup of the instance and the volumes separately. To backup the instance, the recommendation is to create an image, as shown above. For each of the volumes attached, one can create a snapshot or a backup for each of them by following the instructions for volume snapshots or volume backups above.

Single VM and multiple networks and/or security groups

The network configuration and security groups of VMs are not recorded when performing an instance snapshot. If this information needs to be backed up in order to reconfigure the VM after restoration, we recommend storing these values in a file inside the VM. These can be done, e.g. in YAML format, with

openstack server show -f yaml -c addresses -c security_groups <instance> > <file>

From these stored details, the network and security group configuration can be restored. Visit EWC - How to restore backups from VMs to find out more about how to restore instances.

Single VM and SFS

Shared File Services (SFS) cannot be backed up as they are distributed storage solutions that do not live in the same cloud. CONFIRM THIS. The recommended way to backup a VM with a SFS is to unmount the SFS before performing the instance snapshot, in order to avoid possible corruption, and remounting it afterwards. This can be easily done, with admin privileges, with

sudo umount <sfs_mount_point>
...
<backup instance, volumes, etc>
...
sudo mount <sfs_url> <sfs_mount_point>

Multiple VMs, volumes, etc

In order to backup multiple VMs, with any configuration, one can repeat the steps laid out above multiple times in succession. Attention must be when backuping up volumes that are attached to multiple instances or any other possible overlapping situation. For the sake of safety, it is recommended to detach all the volumes and stop all the instances that are to be backed up.

An alternative to repeating all the steps here multiple times is to use the automation script below, where one can set up a configuration file detailing all the instances, volumes, etc, to be backed up, and run the script a single time.

Scheduling backups

It is often quite useful to schedule a backup for a specific time in the future, as well as to repeat that scheduled backup regularly, to ensure the preservation of any changes. In order to prepare a scheduled backup, with or without repetition, one can set up a cron job that will run in the background and perform the necessary operations. A full tutorial on how to use cron is beyond the scope of this page, more details can be found elsewhere, e.g. https://crontab.guru/. Here only a few examples will be shown.

First, it is worth reminding that it it advisable that the creation of snapshots and backups is done in a different machine that the one backing up. Consequently, the cron job should also be run in a different machine. To run a cron job, ensure that cron is installed in the machine, e.g.

 sudo apt update && sudo apt install cron

in Ubuntu distributions, and start the cron services with

sudo systemctl enable cron

Once cron is running, to configure a cron job, one must edit the crontab file with

crontab -e

and add there the command to be executed and the frequency. For example, to create an image snapshot of a VM every Saturday at 12pm, the crontab entry will look like

0 12 * * Sat openstack server image create --name <image_name> <server>

.Note that this job will necessarily fail, since the cron service does not use the environment variables with the OpenStack credentials that are needed to use the OpenStack CLI. Hence, a better solution is to write a shell script that will load the appropriate environment variables and run any operations. Thus, the same example with the correct OS credentials will look like

0 12 * * Sat run_backup_job.sh 2>&1 | logger -t backups

which will run the script run_backup_job.sh and log all the output. The script run_backup_job.sh is then

#! /usr/bin/env bash

for var in $(env | sed -n 's/^\(OS.*\)=.*/\1/p'); do unset "$var"; done
export OS_AUTH_URL=https://keystone.cloudferro.com:5000/v3
export OS_INTERFACE=public
export OS_IDENTITY_API_VERSION=3
export OS_USERNAME="<username>"
export OS_REGION_NAME="WAW3-1"
export OS_PROJECT_ID="<project_id>
export OS_PROJECT_NAME="<project_name>"
export OS_PROJECT_DOMAIN_ID="<domain_id>"
export OS_PASSWORD="<password>"
export OS_CLIENT_ID=openstack
export OS_CLIENT_SECRET=08eb53c3-3cac-4a01-8406-d72f6a70948b
export OS_PROTOCOL=openid
export OS_IDENTITY_PROVIDER=eumetsat_provider
export OS_AUTH_TYPE=v3oidcpassword
export OS_DISCOVERY_ENDPOINT="https://identity.cloudferro.com/auth/realms/Eumetsat-elasticity/.well-known/openid-configuration"

echo "Stopping server"
openstack server stop  TestVM
echo "Backing up server"
sleep 5
openstack server image create --name TestVM_snapshot --wait TestVM
echo "Restarting server"
openstack server start TestVM

which, in addition to creating the snapshot, will stop the server beforehand, and then restart it, waiting for the server to be stopped and image to be created in between.

To run the backups only once, without repetition, it is possible to set up a conditional statement on the script, such as

...
<set up env>
...

COMMAND=run_backup_job.sh
DONE="${COMMAND}.alreadyrun"
   
if [[ -f $DONE ]]; then
  exit 1
fi
touch "$DONE"

...
<run some commands>
...

Alternatively, the same result can be obtained using the at command instead of cron , by running

./run_backup_job.sh | at <some_future_time>

Lastly, when scheduling backups, it is recommended to set up a retention limit, so that no more than a certain number of backups are kept at a given time. One can incorporate this check, and the corresponding deletion of older backups, in the cron script, with

...
<do some backup>
...

retention_count=5
backups=(`openstack image list --property image_type=snapshot --property instance_uuid=<instance_id> --sort updated_at -c ID -f value`)
nbackups=${#backups[@]}
echo "Found $nbackups backups of instance"
if [[ $nbackups -gt $retention_count ]]
then
    let "todelete=$nbackups-$retention_count"
    echo "Backup number over the retention count, will delete the oldest $todelete backups"
    for ((i=0;i<todelete;i++)); do
        echo "Deleting image ${backups[$i]}"
        openstack image delete ${backups[$i]}
    done;
else
	echo "Backup number equal or under retention count, doing nothing"
fi

Note: some of the syntax above is specific to bash . Add SHELL=/usr/bin/env bash to your crontab to ensure cron runs using bash , or change the above script for its equivalent in sh .

Backup automation

The multi-step procedure for creating backups of instances and volumes, including the scheduling and repetition of backups, can been performed in a single step using the script found in the repository

TODO: Link to repository

A simple configuration YAML file detailing the instances and volumes to back up, as well as the start and frequency of the scheduling, can be used to run the script. Instructions on how to run the script can be found in the README file in the repository itself.