Python environments
In order to develop Python scripts, the clusters of the IPSL mesocentre propose Python modules which are Python programming environments including a Python interpreter and a set of packages (programming libraries). If you are not satisfied with the proposed modules, you have the possibility to extend them (read this page for more information) or you can create your own Python environment and choose your packages at the version that suits you. There are two kinds of Python environments: conda environments and Python virtual environments. This page describes how to create an environment and install packages in these two types of environment.
Warning
As for the Python modules, Python scripts that have dependencies on packages installed in a Python environment must always run in that activated environment.
Warning
Conda environments take a lot of disk space. Don't forget that your disk space is restricted (quota). You may extend Python modules thank to this procedure or locate your Python environment in large workspaces.
Tip
As the replication of pure conda environment is easy, we recommand to install your packages exclusively with the command conda install. More information on the replication of Python environments at this page.
Info
The Python modules provided are listed under the category /net/nfs/tools/meso-u20/modules/Python when running the command module avail.
Conda environments
The creation of conda environment requires the installation of a miniconda distribution (more information here). Then the creation and management of conda environment is done with the conda command.
Miniconda installation
- Download the last version of Miniconda installer
When installing, Miniconda asks you to initialize itself. If you choose to do so, it will add some instructions to your ~/.bashrc.
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
chmod +x Miniconda3-latest-Linux-x86_64.sh
./Miniconda3-latest-Linux-x86_64.sh
Environment creation
Environment activation
Package installation
Let's try to install the package called numpy, in a already activated environment.
- Basic (install the last version from the default channel):
conda install numpy - With specific version (note wildcard and logical operator):
conda install 'numpy>=1.20.*' - With specific channel:
conda install -c conda-forge numpy
Warning
The default channel is anaconda. Never install package from unknown channels or package that few people downloaded it: it may be an infected package.
Info
Search for the packages at this page.
Tips
-n <environment_name> option can specifies the targeted environment without activating it.
Tips
Run pip list so as to list the installed packages, even if they were installed by conda.
Other usefull commands
Assuming that a conda environment is activated:
- Deactivate an environment:
conda deactivate - List the environments:
conda env list - Delete an environment:
conda env remove -n <environment_name> - Delete a package of an environment:
conda remove <package_name> - List the packages of an environment:
conda list
Info
More information about conda command at this page.
Tips
-n <environment_name> option can specifies the targeted environment without activating it.
Python virtual environments
Python virtual environments only manage Python packages, unlike Conda. For example, you can't install drivers (cudatoolkit, cudnn, etc.) necessary for some Python packages (Tensorflow, Pytorch, etc.). The following instructions assume that you have Python distribution already installed on the computer where you want to create a Python virtual environment (you may load a Python module).
Environment creation
# Create the environment parent dir, if it is not already done.
mkdir -p "/data/${USER}/virtual_envs"
# Create the virtual environment named myenv.
python -m venv "/data/${USER}/virtual_envs/myenv"
Environment activation
Package installation
pip install -U pip # Update pip.
pip install -U <nom_package1> <nom_package2> # Install or upgrade your packages.
Environment deactivation
Environment deletion
Warning
Never install package from unknown authors or package that few people downloaded it: it may be an infected package.
Warning
Installing packages with the command pip may result in upgrading or downgrading previously installed packages (due to dependencies). Be careful, this can impact your experiences! It can also lead to conflicts. In order to check for them, run pip check and please resolve them as pip will not do it for you!
Info
Find Python packages at this address.
Tips
Run pip list so as to list the installed packages.