Python environments
Introduction
There is many python distributions on IPSL Mesocenter.
Avaible through modules, we recommand to use anaconda distribution for python:
# module avail python
---------------------- /etc/modulefiles/Products ----------------------
python/2.7-anaconda python/2.7.6-canopy-1.3(default)
python/2.7-anaconda44 python/3.4-anaconda3
python/2.7-anaconda50 python/3.6-anaconda50
python/2.7.3-epd7
----------- /net/nfs/tools/meso-sl6/modules-spack/Products ------------
python/3.8-anaconda python/meso-3.8 python/meso-3.9
Best choice now (2022) are
python/meso-3.9 python/meso-3.8 anaconda3-py/2021.05 anaconda3-py/2021.11
python/meso-3.9full anaconda 2021.11 (python 3.9.7) with the following addition- basemap, basemap-data-hires, cartopy, xarray, netcdf4, dask-jobqueue, mpi4py
- cdo, esmtools
- pathlib pipenv, plotly, pydot, scikit-optimize, termcolor, virtualenv, zarr
- pytorch-1.8.2-lts
python/meso-3.8full anaconda 2021.05 (python 3.8.8) with the following addition- xarray, cartopy , mpi4py , netcdf4 , dask-jobqueue ,esmtools, pathlib
- pipenv, plotly, pydot, scikit-optimize, termcolor, virtualenv, zarr
- pytorch-1.8.2-lts
All packages I need are in the distribution
I just do for example : module load python/meso-3.9
To know the exact package list : conda list
I need to add some package and modify version of few package
First , I'm loading the module : module load python/conda-2021.05
to be able to add or modify or add package I need to clone the installation
conda create --name "Replace_by_your_proj_name" --clone="$(which python | xargs dirname)/.."
this command could take time , when it's finish, you need to activate your environment
conda activate Replace_by_your_proj_name
the module load command and conda activate ) must be executed in each session to use your python environment ( interactive or batch )
SafetyError warning are normal because some libraries have been patched by us
I want to download my own conda
Warning , our System is very old and new conda distribution cannot be installed anymore
There is no fix for the installer himself
the last Installers Versions known to work are :
- https://repo.anaconda.com/archive/Anaconda3-2021.05-Linux-x86_64.sh
- https://repo.anaconda.com/miniconda/Miniconda3-py39_4.9.2-Linux-x86_64.sh
The actual one cannot be installed directly on Mesocenter :
- https://repo.anaconda.com/archive/Anaconda3-2021.11-Linux-x86_64.sh
- https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
I prefer small customized distribution managed by yaml file
Customized distribution is very good for production and could be reproduced on any computer
any package version could be freezed in configuration file and package could be added by conda or pip
It's also a good solution if you need to distribute software to other user on other computer
because you are sure they have the same environment and you just need to give them the yaml file
don't load any python module
download a miniconda distribution
wget https://repo.anaconda.com/miniconda/Miniconda3-py39_4.9.2-Linux-x86_64.sh ; bash ./Miniconda3-py38_4.9.2-Linux-x86_64.sh
Welcome to Miniconda3 4.x.xx
In order to continue the installation process, please review the license
agreement.
Please, press ENTER to continue
>>>
===================================
Miniconda End User License Agreement
===================================
[...]
Do you accept the license terms? [yes|no]
[no] >>> **yes**
Miniconda3 will now be installed into this location:
/home/xxxxxx/miniconda3
- Press ENTER to confirm the location
- Press CTRL-C to abort the installation
- Or specify a different location below
Unpacking payload ...
Collecting package metadata (current_repodata.json): done
Solving environment: done
## Package Plan ##
environment location: /home/xxxxxx/miniconda3
added / updated specs:
- _libgcc_mutex==0.1=main
- asn1crypto==1.2.0=py37_0
- ca-certificates==2019.10.16=0
- certifi==2019.9.11=py37_0
- cffi==1.13.0=py37h2e261b9_0
- chardet==3.0.4=py37_1003
- conda-package-handling==1.6.0=py37h7b6447c_0
- conda==4.7.12=py37_0
- cryptography==2.8=py37h1ba5d50_0
- idna==2.8=py37_0
- libedit==3.1.20181209=hc058e9b_0
- libffi==3.2.1=hd88cf55_4
- libgcc-ng==9.1.0=hdf63c60_0
- libstdcxx-ng==9.1.0=hdf63c60_0
- ncurses==6.1=he6710b0_1
- openssl==1.1.1d=h7b6447c_3
- pip==19.3.1=py37_0
- pycosat==0.6.3=py37h14c3975_0
- pycparser==2.19=py37_0
- pyopenssl==19.0.0=py37_0
- pysocks==1.7.1=py37_0
- python==3.7.4=h265db76_1
- readline==7.0=h7b6447c_5
- requests==2.22.0=py37_0
- ruamel_yaml==0.15.46=py37h14c3975_0
- setuptools==41.4.0=py37_0
- six==1.12.0=py37_0
- sqlite==3.30.0=h7b6447c_0
- tk==8.6.8=hbc83047_0
- tqdm==4.36.1=py_0
- urllib3==1.24.2=py37_0
- wheel==0.33.6=py37_0
- xz==5.2.4=h14c3975_4
- yaml==0.1.7=had09818_2
- zlib==1.2.11=h7b6447c_3
The following NEW packages will be INSTALLED:
_libgcc_mutex pkgs/main/linux-64::_libgcc_mutex-0.1-main
asn1crypto pkgs/main/linux-64::asn1crypto-1.2.0-py37_0
ca-certificates pkgs/main/linux-64::ca-certificates-2019.10.16-0
certifi pkgs/main/linux-64::certifi-2019.9.11-py37_0
cffi pkgs/main/linux-64::cffi-1.13.0-py37h2e261b9_0
chardet pkgs/main/linux-64::chardet-3.0.4-py37_1003
conda pkgs/main/linux-64::conda-4.7.12-py37_0
conda-package-han~ pkgs/main/linux-64::conda-package-handling-1.6.0-py37h7b6447c_0
cryptography pkgs/main/linux-64::cryptography-2.8-py37h1ba5d50_0
idna pkgs/main/linux-64::idna-2.8-py37_0
libedit pkgs/main/linux-64::libedit-3.1.20181209-hc058e9b_0
libffi pkgs/main/linux-64::libffi-3.2.1-hd88cf55_4
libgcc-ng pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0
libstdcxx-ng pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0
ncurses pkgs/main/linux-64::ncurses-6.1-he6710b0_1
openssl pkgs/main/linux-64::openssl-1.1.1d-h7b6447c_3
pip pkgs/main/linux-64::pip-19.3.1-py37_0
pycosat pkgs/main/linux-64::pycosat-0.6.3-py37h14c3975_0
pycparser pkgs/main/linux-64::pycparser-2.19-py37_0
pyopenssl pkgs/main/linux-64::pyopenssl-19.0.0-py37_0
pysocks pkgs/main/linux-64::pysocks-1.7.1-py37_0
python pkgs/main/linux-64::python-3.7.4-h265db76_1
readline pkgs/main/linux-64::readline-7.0-h7b6447c_5
requests pkgs/main/linux-64::requests-2.22.0-py37_0
ruamel_yaml pkgs/main/linux-64::ruamel_yaml-0.15.46-py37h14c3975_0
setuptools pkgs/main/linux-64::setuptools-41.4.0-py37_0
six pkgs/main/linux-64::six-1.12.0-py37_0
sqlite pkgs/main/linux-64::sqlite-3.30.0-h7b6447c_0
tk pkgs/main/linux-64::tk-8.6.8-hbc83047_0
tqdm pkgs/main/noarch::tqdm-4.36.1-py_0
urllib3 pkgs/main/linux-64::urllib3-1.24.2-py37_0
wheel pkgs/main/linux-64::wheel-0.33.6-py37_0
xz pkgs/main/linux-64::xz-5.2.4-h14c3975_4
yaml pkgs/main/linux-64::yaml-0.1.7-had09818_2
zlib pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3
Preparing transaction: done
Executing transaction: done
installation finished.
Do you wish the installer to initialize Miniconda3
by running conda init? [yes|no]
[no] >>> **yes**
no change /home/xxxxxx/miniconda3/condabin/conda
no change /home/xxxxxx/miniconda3/bin/conda
no change /home/xxxxxx/miniconda3/bin/conda-env
no change /home/xxxxxx/miniconda3/bin/activate
no change /home/xxxxxx/miniconda3/bin/deactivate
no change /home/xxxxxx/miniconda3/etc/profile.d/conda.sh
no change /home/xxxxxx/miniconda3/etc/fish/conf.d/conda.fish
no change /home/xxxxxx/miniconda3/shell/condabin/Conda.psm1
no change /home/xxxxxx/miniconda3/shell/condabin/conda-hook.ps1
no change /home/xxxxxx/miniconda3/lib/python3.7/site-packages/xontrib/conda.xsh
no change /home/xxxxxx/miniconda3/etc/profile.d/conda.csh
modified /home/xxxxxx/.bashrc
==> For changes to take effect, close and re-open your current shell. <==
If you'd prefer that conda's base environment not be activated on startup,
set the auto_activate_base parameter to false:
conda config --set auto_activate_base false
Thank you for installing Miniconda3!
[/home/xxxxxx/miniconda3] >>>
if you do not answer yes to the last question you need to prepend in your path the directory of the conda command export PATH=/home/xxxxxx/miniconda3/bin:$PATH
create a yaml file like this sample
# name of my project virtualenv
name: MyProj
# needed anaconda channels
channels:
- defaults
- conda-forge
- mro
# package list to install via conda
dependencies:
- jupyterlab=0.35
- matplotlib<3.1
- pandas=0.24
- python=3.7
- scikit-learn=0.20
- rake_nltk
- beautifulsoup4
- psycopg2=2.8
- requests=2.21
- pymongo=3.7
- pytest
- scikit-learn=0.20
- scipy=1.2
- seaborn=0.9
- statsmodels=0.9
- flake8
- openpyxl=2.6
- pip
# package list installed via pip
- pip:
- pycryptodomex==3.8.1
- pytest-mpl==0.10
- pytest-cov==2.6.1
then launch the following command : conda env create -f MyProj.yml
Each time you need to use your distribution you have to do
source activate MyProj
and it is a good practice to exit properly when you are done working in this setting
source deactivate MyProj