Extending an Python module
The Python modules available on IPSL's systems contain hundreds of Python packages. However, you may not find the package you need for your project or the package may not be at the right version. This section describes a procedure to extend an Python module. Extending an Python module is done by overlaying a virtual Python environment on top of the module.
The packages of a Python module are always accessible from the Python virtual environment. Indeed, the packages installed in the Python virtual environment come in addition to or replace those of the module.
This technique is not without restrictions: Python virtual environments only manage Python packages, unlike Conda. For example, you can't install drivers (cudatoolkit, cudnn, etc.) necessary for some Python packages (Tensorflow, Pytorch, etc.). However, if the drivers present in the module satisfy the packages you want to install (list of packages in a module: conda list), the technique described in this section is sufficient. First of all, try this procedure. If it fails, we advise you to create a Conda environment according to your specifications (Read more here).
Warning
Conda environments take more disk space than Python virtual environments. Don't forget that your disk space is restricted (quota). If possible, use Python virtual environments, located in large working spaces.
Python virtual environments
This example shows how to extend the module pangeo-meso/2024.01.22, thanks to the creation of a Python virtual environment named myenv, located in the directory ${HOME}/virtual_envs. From a cluster head node:
module load pangeo-meso/2024.01.22 # Load the module to be extended.
mkdir -p "${HOME}/virtual_envs" # Create the environment parent dir, if it is not already done.
python -m venv --system-site-packages "${HOME}/virtual_envs/myenv" # Create the virtual environment named myenv.
source "${HOME}/virtual_envs/myenv/bin/activate" # Activate the virtual environment.
pip install -U pip # Update pip.
pip install -U <nom_package1> <nom_package2> # Install or upgrade your packages.
Example of the activation of the Python virtual environment named myenv:
De-activation of the current Python virtual environment:
Deletion of the Python virtual environment named myenv:
Warning
Code that has dependencies on packages installed in the myenv environment must always run in an environment where the module extended by myenv and the myenv environment are loaded (example: module load pangeo-meso/2024.01.22) and activated (example: source "${HOME}/virtual_envs/myenv/bin/activate"), respectively. Load the module first then activate the virtual environment.
Warning
Installing packages with the command pip may result in upgrading or downgrading previously installed packages (due to dependencies). Be careful, this can impact your experiences! It can also lead to conflicts. In order to check for them, run pip check and please resolve them as pip will not do it for you!
Info
Find Python packages at this address.