Conda environments in Jupyter ecosystem without pain

Photo by Brett Jordan on Unsplash

Jupyter ecosystem is one of the most popular for doing interactive science these days. And to install the needed dependencies, lots of people are relying on conda as package manager.

This article will describe a reliable method to integrate conda environments with Jupyter tools; aka JupyterLab but also nbconvert (notebook converter), voila (notebook as dashboard), papermill (parametrized notebook),… .

The final goal is to create and deploy voila dashboards with any conda environments installed.

The examples were generated using Linux Debian 10.

JupyterLab installation

After installing conda on our computer, we will install JupyterLab and voila in the base environment:

(base) $ conda install -c conda-forge --yes jupyterlab voila nodejs ipywidgets

To ease dashboard creation in JupyterLab, we will add the voila preview extension and the Jupyter widgets to have interactivity in the dashboard:

(base) $ jupyter labextension install @jupyter-voila/jupyterlab-preview @jupyter-widgets/jupyterlab-manager

If we launch JupyterLab, using

(base) $ jupyter lab

there is one kernel available for notebook; it is called Python 3. And we are able to execute a notebook with widgets and show its voila preview.

Create a new environment

A good practice is to create a conda environment for each project you are working on to help maintenance and reproducibility. So we will create a new conda environment called project1 that uses Python 3 and the Jupyter widgets. Don’t forget to add ipykernel otherwise Jupyter won’t know about it.

(base) $ conda create -n project1 python=3 ipykernel ipywidgets

Now how can we use that environment in JupyterLab. We could install JupyterLab in the project1 environment. But if we do so, we will have to install JupyterLab and its extensions in each environment we are creating. Fortunately the package nb_conda_kernels will solve that problem by providing the conda environment as Jupyter kernels.

Be sure to install it in the base environment where JupyterLab got installed.

(base) $ conda activate base
(base) $ conda install -c conda-forge nb_conda_kernels

Now in JupyterLab, we see two environments (Python [conda env: root]* is a duplicate of Python 3). And if we pick the new project1 environment as kernel for the notebook, the voila preview is picking it too. So far so good.

To share a dashboard with our friends, we need to start a voila server by executing the command:

(base) $ voila Demonstration.ipynb 

And now the unexpected happens, the standalone dashboard is using the base environment and not project1.

Looking at voila log messages, we can see that voila was not able to find the kernel conda-env-project1-py. So it used the default python3 kernel.

[Voila] Using /tmp to store connection files
[Voila] Storing connection files in /tmp/voila_1wkja6ak.
[Voila] Serving static files from /.../lib/python3.8/site-packages/voila/static.
[Voila] Voilà is running at:
http://localhost:8866/
[JupyterApp] WARNING | Config option `kernel_spec_manager_class` not recognized by `JupyterApp`.
[Voila] WARNING | Could not find a kernel named 'conda-env-project1-py', will use 'python3'
[Voila] Kernel started: e7a34317-d4a7-43cd-bc8a-6dbe38850c42

This is really bad because the project1 environment may contain packages not available in the base one or having a different version.

Configure nb_conda_kernels

To tackle that problem, we will use a new feature of nb_conda_kernels introduced in version 2.3.0: the dynamic generation of Jupyter kernelspecs.

To activate that feature, we need to edit the jupyter configuration file. Open the configuration file:

$ nano $HOME/.jupyter/jupyter_config.json

Add the following configuration:

{
"CondaKernelSpecManager": {
"kernelspec_path": "--user"
}
}

To check the configuration we execute first:

(base) $ python -m nb_conda_kernels list[ListKernelSpecs] WARNING | Config option `kernel_spec_manager_class` not recognized by `ListKernelSpecs`.
[ListKernelSpecs] Installed kernelspec conda-root-py in /home/fcollonval/.local/share/jupyter/kernels/conda-root-py
[ListKernelSpecs] Installed kernelspec conda-env-project1-py in /home/fcollonval/.local/share/jupyter/kernels/conda-env-project1-py
[ListKernelSpecs] [nb_conda_kernels] enabled, 2 kernels found
Available kernels:
conda-env-project1-py /home/fcollonval/conda3/envs/project1/share/jupyter/kernels/python3
conda-root-py /home/fcollonval/conda3/share/jupyter/kernels/python3

Then we execute:

(base) $ jupyter kernelspec list[ListKernelSpecs] WARNING | Config option `kernel_spec_manager_class` not recognized by `ListKernelSpecs`.
Available kernels:
conda-env-project1-py /home/fcollonval/.local/share/jupyter/kernels/conda-env-project1-py
conda-root-py /home/fcollonval/.local/share/jupyter/kernels/conda-root-py
python3 /home/fcollonval/conda3/share/jupyter/kernels/python3

If the configuration is correct, they both should list all conda environments as shown above. In particular they list conda-env-project1-py, the kernel to be used by the dashboard.

And indeed, if we launch now the standalone dashboard (with voila Demonstration.ipynb), the expected python is now used:

To put it in a nutshell

To summarize, you need three actions to access your conda environments in Jupyter tools:

  • Install ipykernelin all environments (for non Python environment look on the internet to figure out the equivalent package)
  • Install nb_conda_kernels (version ≥2.3.0) in the base environment along side Jupyter tools like JupyterLab or voila.
  • Set the following configuration in $HOME/.jupyter/jupyter_config.json:
{
"CondaKernelSpecManager": {
"kernelspec_path": "--user"
}
}

Bonus

Here is a tip for the courageous reaching this section. nb_conda_kernels allows to customized the environment displayed name in JupyterLab by tuning the name_format setting. With the following example

{
"CondaKernelSpecManager": {
"kernelspec_path": "--user",
"name_format": "{kernel} ({environment})"
}
}

we will get

conda is developed and maintained by Anaconda, Inc.
Jupyter is a trademark of the Jupyter Project

--

--

--

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

All the While

Declutter Your Hierarchy and Spark Joy in Your Projects

High performance persistent queue using db-scheduler

How to check for an active Internet connection on iOS or macOS?

Referral Program with Rewards up to $10k per Project.

The SPROUT STORY: Opening a Wide Array of Opportunities for Underprivileged Youth — Impact…

Functions in R

Rebuttal of an Agile Development Naysayer

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Frédéric Collonval

Frédéric Collonval

More from Medium

Mapping scRNA-seq data using Cell Ranger

Python’s powerful “yield” keyword

What has COVID been like in San Francisco?

Fetching subsets of VCF files, a DIY approach