Jupyter notebooks batch execution
Basic batch execution of notebooks
With Jupyter, comes a very interesting command: jupyter-nbconvert (see here for another kind of installation). This command allows the conversion of a notebook file (*.ipynb) into a python, pdf, html, latex, rst, etc. script and especially the execution of a notebook (for a list of all formats, type jupyter-nbconvert --help).
The Nbconvert command is very interesting if you want to run a notebook as a batch job on the mesocenter clusters. Knowing that cluster job orchestrators only run scripts, nbconvert is a real deal: it saves you to manage a scripted version of your notebook, just for its execution on clusters (management that inevitably comes with update problems).
The command line is simple by default (for the options, visit this page and visit this page for the templating system). Assuming you have the command in your module environment, the example below runs the notebook named myfile.ipynb and the result is saved in a file named myfile.nbconvert.ipynb. More information at this page.
All you have to do, is to integrate this command line into a bootstrap Bash script and submit it as a batch job to the cluster scheduler.
Info
The option --ExecutePreprocessor.timeout=-1 disables the cell execution timeout whose default value is 30 seconds.
Info
If you want the result to be saved directly in the notebook, add the --clear-output option. But it is strongly recommended to use the --inplace option instead to avoid mixing the result with the previous one.
Convert notebook into a html page
Distribute your notebook ipynb file is nice but distribute it with a html cover page is nicer: it lets people to read it without running Jupyter Notebook.
Conditional execution of notebook cells
Conditional execution of notebook cells is possible using the nbconvert command (see above for its introduction). The first step is to mark the cells that you want to ignore during the execution of the notebook, using the Jupyter marking system:
Save the notebook, then run the notebook with the following command line:
jupyter-nbconvert --to notebook --ExecutePreprocessor.timeout=-1 --TagRemovePreprocessor.enabled=True --TagRemovePreprocessor.remove_cell_tags 'NBCONVERT_IGNORED' --execute myfile.ipynb
Warning
Never use the --inplace and --clear-output options unless you want to lose the marked cells forever!!!
Info
The NBCONVERT_IGNORED marker does not matter but it must be consistent with the command line.
Info
Nbconvert has a lot of options. Combined with the Jupyter marking system, it is possible to emulate conditional code generation (a bit like the C language preprocessor).