...
| Info | ||
|---|---|---|
| ||
Conda is made available at ECWMF through the module system, so you don't need to install it to use it on HPCF. However, you may also want to install it on your work/personal computer to create and organise your environments. The simplest way is to download the Miniconda distribution:
Or even on Windows if you're feeling wild: Miniconda3-latest-Windows-x86_64.exe Just run the script to install Conda. It will update your shell config so you'll need to reload your shell session to access it. For linux (similar approach on MacOS):
If that works, you'll see a (base) at the start of your prompt, which tells us we are in the "base" conda environment. |
...
Let's start with a basic environment containing Python and Numpy:
|
You can then load it using:
|
The environment will be stored next to where Conda is installed (or by default in a .conda directory if you can't write to the Conda directory). To install the environment in a dedicated path, you can use the following:
|
and load it by specifying the full path:
|
Once the environment is loaded, you can install more packages:
|
To list the packages in your environment:
|
Once you obtain a stable environment, you can export it to a yaml file:
|
You can then use this file to reproduce exactly the environment, share it with other users, use it as a base for other environments, etc. To create a new environment from a yaml file, run the following command:
|
To unload your environment, just run:
|
To remove your environment:
|
Finally, to see the list of available environments:
|
See Conda documentation and tutorials for more ways to create and manage environments.
...
Going back to our first environment with only python and numpy installed. When you install the following packages:
|
you will see that Conda will also install all the packages dependencies. It will automatically solve the conflicts between dependencies. This is why if you add another package to the list above, you may end up with a completely different environment, as the common dependencies between the two list of packages may be different.
You can also use specific version (Conda will try to get the latest by default):
|
This is perfectly valid but this will make Conda's life more difficult as it will have to solve a more constrained dependencies problem. You will see that constraining too much the environment will quickly give you conflicts, so only specify the versions you really need.
To remove a package, simply use:
|
install
|
This will also remove all the dependencies.
To search for a package:
|
For get more information a package (dependencies, timestamp, etc.):
|
This last command can be very useful to debug dependencies conflicts.
...
When installing packages using the commands above, you will use the default channels (Anaconda if you downloaded it using Miniconda or ECMWF channels on ECMWF HPC). On your personal/work machine (not on the HPC), it is recommended to switch to conda-forge (https://conda-forge.org/docs/user/introduction.html), which contains many scientific packages and is more suited to our needs.
This can be done at the command line level by specifying the channel by hand:
|
Or add it permanently to your list of channels using:
|
It is recommended to stick to one channel to avoid conflicts. Only use multiple channels if you know they are compatible with each other.
On the ECMWF HPC infrastructure, we use dedicated default channels, which are based on conda-forge. It is recommended to stick with those channels. More information, see Conda at ECMWF.
Conda and pip
Some packages are not available on Conda but can be accessible through pip. Conda and pip are compatible and users are free to activate their Conda environments and install packages through pip as well. Users should be cautious by doing so as it could lead to dependencies issues. It is always best to try to install packages through Conda whenever possible. Users can also use pip to install their own packages, rather than creating a Conda recipe, which requires more work.
The best practice is to first install all the base packages using Conda and then install on top of that the remaining packages using pip.
...
Useful links
Introduction
https://towardsdatascience.com/introduction-to-conda-virtual-environments-eaea4ac84e28
https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html
https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html
Cheat sheet
Building your own recipe
Conda: how to create recipes and channels
https://docs.conda.io/projects/conda-build/en/latest/concepts/recipe.html#more-information
https://docs.conda.io/projects/conda-build/en/latest/resources/define-metadata.html
Conda config
https://docs.conda.io/projects/conda/en/latest/commands/config.html
https://docs.conda.io/projects/conda/en/latest/user-guide/configuration/use-condarc.html