Conda environments in Jupyter ecosystem without pain

5 min readDec 13, 2020

Jupyter ecosystem is one of the most popular for doing interactive science these days. And to install the needed dependencies, lots of people are relying on conda as package manager.

This article will describe a reliable method to integrate conda environments with Jupyter tools; aka JupyterLab but also nbconvert (notebook converter), voila (notebook as dashboard), papermill (parametrized notebook),… .

The final goal is to create and deploy voila dashboards with any conda environments installed.

The examples were generated using Linux Debian 10.

JupyterLab installation

After installing conda on our computer, we will install JupyterLab and voila in the base environment:

(base) $ conda install -c conda-forge --yes jupyterlab voila nodejs ipywidgets

To ease dashboard creation in JupyterLab, we will add the voila preview extension and the Jupyter widgets to have interactivity in the dashboard:

(base) $ jupyter labextension install @jupyter-voila/jupyterlab-preview @jupyter-widgets/jupyterlab-manager

If we launch JupyterLab, using

(base) $ jupyter lab

there is one kernel available for notebook; it is called Python 3. And we are able to execute a notebook with widgets and show its voila preview.

Create a new environment

A good practice is to create a conda environment for each project you are working on to help maintenance and reproducibility. So we will create a new conda environment called project1 that uses Python 3 and the Jupyter widgets. Don’t forget to add ipykernel otherwise Jupyter won’t know about it.

(base) $ conda create -n project1 python=3 ipykernel ipywidgets

Now how can we use that environment in JupyterLab. We could install JupyterLab in the project1 environment. But if we do so, we will have to install JupyterLab and its extensions in each environment we are creating. Fortunately the package nb_conda_kernels will solve that problem by providing the conda environment as Jupyter kernels.

Be sure to install it in the base environment where JupyterLab got installed.

(base) $ conda activate base
(base) $ conda install -c conda-forge nb_conda_kernels

Now in JupyterLab, we see two environments (Python [conda env: root]* is a duplicate of Python 3). And if we pick the new project1 environment as kernel for the notebook, the voila preview is picking it too. So far so good.

To share a dashboard with our friends, we need to start a voila server by executing the command:

(base) $ voila Demonstration.ipynb

And now the unexpected happens, the standalone dashboard is using the base environment and not project1.

Looking at voila log messages, we can see that voila was not able to find the kernel conda-env-project1-py. So it used the default python3 kernel.

[Voila] Using /tmp to store connection files
[Voila] Storing connection files in /tmp/voila_1wkja6ak.
[Voila] Serving static files from /.../lib/python3.8/site-packages/voila/static.
[Voila] Voilà is running at:
http://localhost:8866/
[JupyterApp] WARNING | Config option `kernel_spec_manager_class` not recognized by `JupyterApp`.
[Voila] WARNING | Could not find a kernel named 'conda-env-project1-py', will use  'python3'
[Voila] Kernel started: e7a34317-d4a7-43cd-bc8a-6dbe38850c42

This is really bad because the project1 environment may contain packages not available in the base one or having a different version.

Configure nb_conda_kernels

To tackle that problem, we will use a new feature of nb_conda_kernels introduced in version 2.3.0: the dynamic generation of Jupyter kernelspecs.

To activate that feature, we need to edit the jupyter configuration file. Open the configuration file:

$ nano $HOME/.jupyter/jupyter_config.json

Add the following configuration:

{
  "CondaKernelSpecManager": {
    "kernelspec_path": "--user"
  }
}

To check the configuration we execute first:

(base) $ python -m nb_conda_kernels list[ListKernelSpecs] WARNING | Config option `kernel_spec_manager_class` not recognized by `ListKernelSpecs`.
[ListKernelSpecs] Installed kernelspec conda-root-py in /home/fcollonval/.local/share/jupyter/kernels/conda-root-py
[ListKernelSpecs] Installed kernelspec conda-env-project1-py in /home/fcollonval/.local/share/jupyter/kernels/conda-env-project1-py
[ListKernelSpecs] [nb_conda_kernels] enabled, 2 kernels found
Available kernels:
  conda-env-project1-py    /home/fcollonval/conda3/envs/project1/share/jupyter/kernels/python3
  conda-root-py            /home/fcollonval/conda3/share/jupyter/kernels/python3

Then we execute:

(base) $ jupyter kernelspec list[ListKernelSpecs] WARNING | Config option `kernel_spec_manager_class` not recognized by `ListKernelSpecs`.
Available kernels:
  conda-env-project1-py    /home/fcollonval/.local/share/jupyter/kernels/conda-env-project1-py
  conda-root-py            /home/fcollonval/.local/share/jupyter/kernels/conda-root-py
  python3                  /home/fcollonval/conda3/share/jupyter/kernels/python3

If the configuration is correct, they both should list all conda environments as shown above. In particular they list conda-env-project1-py, the kernel to be used by the dashboard.

And indeed, if we launch now the standalone dashboard (with voila Demonstration.ipynb), the expected python is now used:

To put it in a nutshell

To summarize, you need three actions to access your conda environments in Jupyter tools:

Install ipykernelin all environments (for non Python environment look on the internet to figure out the equivalent package)
Install nb_conda_kernels (version ≥2.3.0) in the base environment along side Jupyter tools like JupyterLab or voila.
Set the following configuration in $HOME/.jupyter/jupyter_config.json:

{
  "CondaKernelSpecManager": {
    "kernelspec_path": "--user"
  }
}

Bonus

Here is a tip for the courageous reaching this section. nb_conda_kernels allows to customized the environment displayed name in JupyterLab by tuning the name_format setting. With the following example

{
  "CondaKernelSpecManager": {
    "kernelspec_path": "--user",
    "name_format": "{kernel} ({environment})"
  }
}

we will get

conda is developed and maintained by Anaconda, Inc.
Jupyter is a trademark of the Jupyter Project