Jupyter Job

This page explains you how to run a JupyterLab within an batch cluster job and an interactive cluster job.

Within batch job (recommended)

Overview

Connect to a cluster.
Run Jupyterlab in batch job.
Get the Jupyterlab URL.
Get the cluster node name where the batch job is running.
Set a SSH tunnel.
Get JupyterLab on a local web browser.

For step 1, refer to the cluster documentation.

Run Jupyterlab in batch job

While connected to Spirit or SpiritX HPC cluster, submit the Jupyterlab job:

sbatch <path to the parent directory of jupyterlab.slurm>/jupyterlab.slurm

The jupyterlab.slurm containt:

#!/bin/bash


#SBATCH --partition=zen4
#SBATCH --job-name=jupyterlab
#SBATCH --time=8:00:00
#SBATCH --cpus-per-task 1
#SBATCH --mail-user=xxx@ipsl.fr
#SBATCH --mail-type=all
#SBATCH --error=jupyterlab.log
#SBATCH --output=jupyterlab.log

module purge
module load pangeo-meso/2024.01.22 # or any Python module that suits your application.

PORT=8888 # <number between 10000 and 15000>
jupyter lab --no-browser --ip=0.0.0.0 --port=${PORT}

returned_code=$?
echo "> script completed with exit code ${returned_code}"
exit ${returned_code}

Info

Don't forget to set your email address, the job specs (memory and cores) and the Python module as required.

Warning

It is not possible to allocate the same port number twice on the same machine. Choose another port number (variable PORT) if Jupyter refuses to start because of the port. Afterwards, keep the same number in the commands that follow.

Tip

If #SBATCH --chdir is not provided, Jupyterlab root directory is the directory where you ran sbatch. For example, if you run sbatch <path to the parent directory of jupyterlab.slurm>/jupyterlab.slurm from $HOME, $HOME is the Jupyterlab root directory. You can change the Jupyterlab root directory providing --chdir instruction (e.g., #SBATCH --chdir=/home/sgardoll/cyclone_detection).

Get the Jupyterlab URL

Run the following line and wait until Jupyterlab gives you its URL (the one starting with http://127.0.0.1):

tail -f <path to the parent directory of jupyterlab.slurm>/jupyterlab.log

Info

Press CTRL + c so as to quit tail -f.

Get the cluster node name where the batch job is running.

Run the following command, the name of the node appears in the last column. Let's say that the name of the node is spirit64-01.

squeue --me | grep jupyter

Set a SSH tunnel

From another terminal on your machine, run the following lines. Don't forget to set PORT and HOSTNAME accordingly to the previous section:

PORT=<the chosen port number>
HOSTNAME=<the allocated machine fully qualified address>
ssh -N -L ${PORT}:${HOSTNAME}:${PORT} <cluster_login>@<the fully qualified address of the cluster head node>

Example:

PORT=8888
HOSTNAME='spirit64-01'
ssh -N -L ${PORT}:${HOSTNAME}:${PORT} sgardoll@spirit2.ipsl.fr

Info

There are no space characters in the expression ${PORT}:${HOSTNAME}:${PORT}

Info

The command ssh -N -L does not return anything. That is a correct behavior.

Get JupyterLab on a local web browser

Open a local web browser and past the URL address that you noticed when starting JupyterLab (the URL starting with: http://127.0.0.1).

Info

Keep terminals open until the end of your Jupyterlab session.

Info

Shutting down the Jupyterlab (menu file::shut down) ends the batch job.

Within interactive job

Overview

Connect to a cluster.
Request an interactive job.
Load a Python module that contains JupyterLab or Jupyter Notebook.
Run JupyterLab in interactive job.
Get the cluster node name where the batch job is running.
Set a SSH tunnel.
Get JupyterLab on a local web browser.

From steps 1 to 3, refer to the cluster documentation.

Warning

At the end of your work session, close all your terminals connected to the cluster (press CTRL and C several times) in order to free the allocated resources (because simply leaving the notebook does not stop your interactive job). If any sessions remain blocked, you can cancel all your jobs with a specific job manager command (see your cluster documentation).

Run JupyterLab in interactive job

Copy and past this line in the interactive job shell, don't forget to set the port number:

PORT=<number between 10000 and 15000>
hostname -f
jupyter lab --no-browser --ip=0.0.0.0 --port=${PORT}

Warning

It is not possible to allocate the same port number twice on the same machine. Choose another port number (variable PORT) if Jupyter refuses to start because of the port. Afterwards, keep the same number in the commands that follow.

Info

Note the connection URL address displayed by Jupyter in the terminal that starts with: http://127.0.0.1.

Info

Note the output of the command hostname -f: it is "the allocated machine fully qualified address" underneath.