Python 2 support
Only Python 3 is provided, as Python 2 was officially declared end-of-life by the January 1st, 2020.
While Python 3 can be found in the system /usr/bin/python3
, this version does not come with many of the extra modules you may need. In those cases, you may want to use the python3 module or the conda module (still under development).
The python3 module
This is how Python has been made available on previous ECMWF computing platforms. Newer versions are provided twice a year, and they come with more than 200 extra Python packages. You can get the default version by:
module load python3
virtual environments
If you need to customise your Python environment, you may create a virtual environtment based on the installations provided. This may be useful if you need to use a newer version of a specific python package, but still want to benefit from the rest of the managed Python environment:
module load python3 mkdir -p $PERM/venvs cd $PERM/venvs python3 -m venv --system-site-packages myvenv
Then you can activate it when you need it with:
source $PERM/venvs/myvenv/bin/activate
The conda module
We also provide a conda module which will allow you to create your own conda environments tailored to your requirements. You can start using conda with:
module load conda
modules and conda incompatibilities
While conda may be seen as a way to set up custom Python environments, it also manages software beyond that, installing other packages and libraries not necessarily related to Python itself.
Because those may conflict with the software made available through modules, loading the conda module effectively disables all the other modules that may be loaded in your environment.
See HPC2020: Conda for more information.
Containerised conda environments with Tykky
Running conda directly on HPCF may feel very slow due to the nature of the HPCF filesystems, and may contribute to an overall degradation in their performance for all users. We recommend the use of Tykky to containerise your conda environment, which should help you in getting better responsiveness and help alleviate the pressure on the filesystems.
modules and tykky incompatibilities
Like conda, tykky environments may conflict with the software made available through modules. Therefore, to keep your environment consistent, loading the tykky module effectively disables all the other modules that may be loaded in your environment.
See HPC2020: Containerised software installations with Tykky for more information
4 Comments
Christopher Roberts
How do I re-enable modules after unloading conda? In the example below I cannot return to the previous state after loading and unloading conda.
Xavier Abellan
Best strategy is to reset to default with:
Alternatively, because conda is presented as a prgenv, you can just go back to one of them. For example:
Christopher Roberts
Thanks Xavi - is it possible to add this info to the message received when loading conda? e.g. change to
"
Using conda effectively disables any other loaded modules. [Use "module reset" to return to the default environment before loading other modules.]"
Xavier Abellan
Thanks for the feedback. I have added an extra sentence on the message.