...

• You must have a valid ECMWF account with privileges to access HPCF or ECS. If you only have access to ECS, you may need to skip certain exercises involving HPCF.
• You must have 2 Factor Authentication enabled with TOTP.
• You must be able to connect with at least one of the following methods:
  • Using the Virtual Desktop Infrastructure (VDI). See the corresponding documentation to get started.
  • Using Teleport SSH from your end user device own computer.

Logging into Atos HPCF and ECS

...

1. Try to run the command below. Why does it fail? Can you make it work without installing pandas yourself?

   No Format
   $ python3 -c "import pandas as pd; print(pd.__version__)" Traceback (most recent call last): File "<string>", line 1, in <module> ModuleNotFoundError: No module named 'pandas'

   
   

   Expand
   title Solution
   The system Python 3 installation is very limited and does not come with many popular extra packages such as pandas. You may use the Python3 stack available in modules, which comes with almost 400 of those extra packages : No Format module load python3 After that, if you repeat the command it should complete successfully and print pandas version. No Format python3 -c "import pandas as pd; print(pd.__version__)"

   
   

2. You need to use the latest version of pandas to run a given application. What can you do (without using conda)?

   Expand
   title Solution
   In that case you could use pip to install it yourself. However, installing it directly into your user environment is highly discouraged since it may interfere with other applications you may run or after default software updates on the system side. Instead, for small additions to the default environment it is much more robust to use a python virtual environment. In this case, you may create a virtual environment based on the installations provided, and just add the new version of pandas: No Format module load python3 mkdir -p $PERM/venvs cd $PERM/venvs python3 -m venv --system-site-packages myvenv Then you can activate it only when you need it with: No Format source $PERM/venvs/myenv/bin/activate Note that we used $PERM/venvs as the location of these virtual environments, but you may decide to put them in another location.  With the environment activated, you can now install the new version of pandas: No Format pip install -U pandas Then you can rerun the version command to check you got the latest No Format python3 -c "import pandas as pd; print(pd.__version__)" When you have finished with your environment, you can deactivate it with: No Format deactivate

   
   

3. You may also use conda to create your own software stack with python packages and beyond. In order to use conda, you can load the corresponding module:

   No Format
   module load conda

   What happened?

   Expand
   title Answer
   While conda may be seen as a way to set up custom Python environments, it also manages software beyond that, installing other packages and libraries not necessarily related to Python itself. Because those may conflict with the software made available through modules, loading the conda module effectively disables all the other modules that may be loaded in your environment. You have seen how the module system may have disabled a number of modules. You can also check it by running: No Format module list You would then need to install everything you need to run your application or workflow in your conda environment. If you want to go back to the previous environment without conda but with all the other modules, the recommended way is to reset the environment and then load explicitly all the necessary modules again No Format module reset module load python3

   
   

4. Create your new conda environment with latest pandas in it. Check the version Hint: you can also use mamba to speed up the environment creation process

   Expand
   title Solution
   In that case you could use pip to install it yourself. However, installing it directly into your user environment is highly discouraged since it may interfere with other applications you may run or after default software updates on the system side. Instead, for small additions to the default environment it is much more robust to use a python virtual environment. In this case, you may create a virtual environment based on the installations provided, and just add the new version of pandas: No Format mamba create -n mypandas -c conda-forge python pandas conda activate mypandas python3 -c "import pandas as pd; print(pd.__version__)"

   
   

...