Tykky is a set of tools developed at CSC which make software installations on HPC systems easier and more efficient using Apptainer containers. Its main use cases are Python-based conda or virtual environments, and any software or application with a great number of files that when installed on a shared or parallel filesystem may perform very poorly. Tykky will encapsulate those installations in containers, and generate the necessary wrappers so that installed software can be used (almost) as if it was not containerised, improving startup times and reducing the overall load on the filesystems. 

To start using Tykky you must first load its module:

module load tykky

Containerising a conda environment

You can create a containerised conda environment from an environment file. You can either create it manually or from an existing conda environment with:

conda env export -n <target_env_name> > env.yml

An example env.yml file may look like:

channels:
    - conda-forge
dependencies:
    - python
    - ipykernel
    - cf-units
    - eccodes
    - pip
    - pip:
      - earthkit

The example would install Python and eccodes from the conda-forge channel, as well as earthkit through pip. You may also pin specific versions if desired.

Once your environment file is ready, you can use Tikky to perform the installation for you. You must decide where to install it, but use of HPCPERM is strongly recommended. Use of HOME, PERM or SCRATCH for such installations is discouraged for performance reasons.

conda-containerize new --mamba --prefix <install_dir> env.yml

If your environment is a complex one, and you are performing the installation from a standard interactive login session, you may hit some limitations in the amount of memory or temporary storage. If that is the case, you can either submit the installation as a batch job specifying higher memory or storage requirements, or do it interactively with ecinteractive requesting the necessary resources.

For example, a typical AI/ML environment may require more than 20 GB of memory and 30 GB of local TMPDIR space. You may get such session with ecinteractive

ecinteractive -c 4 -m 20 -s 30


When the installation is complete, you will find all the executables in your environment in <install_dir>/bin.

Using your containerised environment

You may activate your new environment with the convenience tykky activate function, very similar to how you would do it with conda:

tykky activate <install_dir>

When you are done, you can also deactivate it with:

tykky deactivate

Alternative use

You may alternatively call them directly using the full path, or add it to your PATH to make them available as if you were activating the environment:

export PATH="<install_dir>/bin:$PATH"

Your containerised environment may contain a number of executables such as python, which may mask others you may have previously loaded in your environment.

Updating an existing containerised conda environment

If you wish to modify an existing installation, you must create a bash script containing the commands you want to execute to update the installation. They will be executed inside the container, with the environment activated, when you run:

conda-containerize update <install_dir> --post-install update.sh

Here is an example of an update script, which would install metview-batch on top of the existing environment

#!/bin/bash
mamba install -y metview-batch

Removing a containerised conda environment

If you no longer need an existing containerised environment, you may simply delete the top level installation directory

rm -rf <install_dir>

Integration with JupyterHub

You can expose your containerised conda environment as an additional kernel in ECMWF's JupyterHub service.  To do that, first make sure you have python and ipykernel installed in your environment, and then you can create the kernel with:

<install_dir>/bin/python3 -m ipykernel install --user --name=<my_containerised_env_name>