ECMWF offers the possibility of using a JupyterLab environment to authorised users. The options available to you will depend on the privileges granted to your user account.

JupyterLab on HPCF or ECS

JupyterHub offers a novel way to access HPCF or ECS services, connecting you to an interactive JupyterLab session where you can run your Jupyter notebooks, manage your files, and much more.

JupyterLab would effectively run within a job on the Atos HPCF or ECS with dedicated resources on an allocated node for the job, and would allow you to conveniently interact with it from your browser without the need of any other connection to the HPCF using other methods. 

First use: enabling access to your account from JupyterHub

Before you can start using this service, you need to enable the access from JupyterHub into your account on the target platform.

A unique SSH key pair is created for this purpose, and you will need to append the generated public SSH key into your ~/.ssh/authorized_keys in your ECMWF HOME space.

You will need to access  Atos HPCF,  ECS or VDI services via other means. From a terminal in one of those systems, you can then:

$ grep -q $USER@jupyter ~/.ssh/authorized_keys || echo "ssh-ed25519 AAAA...." >> ~/.ssh/authorized_keys

Note you should replace "ssh-ed25519 AAAA...." by the actual value of your SSH key that appears in the JupyterHub portal.

Configuring your JupyterLab session

Select the desired profile in the profile dropdown:

Configure the resources and the time allocated for your session:

Configure the project account where your session will be accounted for:

You can also decide wether to use the default version of the Jupyter and Python stack or a newer one with

When you are ready, click Start

You may need to wait between 30 seconds and several minutes for the system to schedule your new job and connect to your new JupyterLab session

After a few minutes, your session will be up and running. Take note of the finishing time at the top right corner of the page. Your session will be terminated automatically once that time is reached.

Stopping your JupyterLab

To stop your JupyterLab session, you may go to the top bar menu, and then click on File - Hub Control Panel

This will bring you back to your JupyterHub Home, from which you will be able to Stop your server (red button) or connect back to it (blue button)

JupyterHub and ecinteractive Jupyter sessions

Since it is also possible to start Jupyter sessions with ecinteractive, you should avoid running two JupyterLabs at the same time with each tool. They run on the same platform using the same HOME filesystem, so you may run into unexpected behaviour.

Accessing other filesystems beyond your HOME

Your Jupyter instance will always start with HOME as your top level directory. In order to access any other of your storage spaces in the Atos HPCF or ECS, you may create soft links to those spaces from your HOME directory. You can do that from a terminal:

for space in perm hpcperm scratch; do ln -snf /$space/$USER ~/$space; done

You may also add those and other frequently accessed places to your favourites:

Leveraging the ECMWF software stack: using modules

You can leverage the existing software stack on the systems with modules with the Lmod extension in Jupyterlab:

Using your own environments

To use your own conda environment as a kernel for Jupyter notebook you will need to have ipykernel installed in the conda environment before starting your Jupyter instance. ipykernel can be installed with:

conda activate myCondaEnv
conda install ipykernel
python3 -m ipykernel install --user --name=myCondaEnv

If you want to make your own Python virtual environment visible in Jupyterlab, the steps are similar:

source myVenv/bin/activate
pip3 install ipykernel
python3 -m ipykernel install --user --name=myVenv

To remove your personal kernels from Jupyterlab once you don't need them anymore, you could do so with:

jupyter kernelspec uninstall myKernelName

Troubleshooting

500: Server internal error

In rare occasions you may this error when spawning a new JupyterLab instance or reconnecting to an existing one. In this case, you will need to manually cancel the failed instance and start a new one. 

You may go to the JupyterHub home at https://jupyterhub.ecmwf.int/hub/home and click on the red button "Stop My Server". After a few seconds, you will be able to start a new one with the desired settings.