Skip to content

Python containerized environment

Introduction

Our contenairization tool is apptainer (https://apptainer.org)

Containerized python environment have many adventages

  • Only one file on our filesystems ( files number quota problems
  • after build one container cannot be modified (reproductibility). If you need to add package container must be rebuilded
  • container could be run on any linux machine ( you just need to have apptainer application in local )
  • you just need to copy one file to use it on other machine or to distribute your environment to other users
  • many HPC center use singularity/apptainer containerization(CINES/IDRIS) or have tool to convert(TGCC)

but could introduce some small difficulties

  • At runtime the only host filesystem viewed is USER home running the container, so you need to predefine which filesystems you need to see in the container
  • you need to write a file definition of your python environment to build your container

Building a micromamba (mamba/conda environment) container for apptainer

you need two files: - one to build the container (micromanba.def) - one to build the mamba environment (environment.yml)

File to build the container

this file should be suffisant for most of usage (this container is based on debian 13)

micromamba.def 

Bootstrap: docker
From: mambaorg/micromamba:latest

%files
environment.yml /opt

%post

    apt -y update
    apt -y upgrade
    apt -y install wget curl 
    apt -y clean    

    micromamba install  -y -n base -f /opt/environment.yml

    micromamba clean --all --yes

%labels
    EnvironmentFile: /opt/environment.yml

%help
    This container containes a Python environment created
    from an environment file.

    To run Python from the environment:

        apptainer run my_environment.sif python

    To see the environment file used to create the environment:

        apptainer run my_environment.sif cat /opt/environment.yml

    To see what packags are in the environment:

        apptainer run micromamba list

file to build the python environment

This is only a sample for a specific application you need to adjust your needed package inside in the environment.yml https://mamba.readthedocs.io/en/latest/user_guide/mamba.html#id2 

name: base
channels:
  - conda-forge
dependencies:
  - python
  - pycontrails

building the container

command to build the container is 

apptainer  build  my_environment.sif micromamba.def

but this cannot be made on headnode on spirit/spiritx cluster, use this insteed 

user@spirit ~> srun --pty -p zen16 apptainer build my_environment.sif micromamba.def
weill@spirit1:~$ ls -lh my_environment.sif 
-rwxr-xr-x 1 user group 904M Sep 11 13:41 my_environment.sif

Running the container

standard command tu run a container

apptainer run <name>.sif