Author: Jens Henrik Göbbert

Index

Create your own Jupyter Kernel with pyenv

Often the standard kernel do not provide all features you need for your work. This might be that certain modules are not loaded or packages are not installed. With your own kernel you can overcome that problem easily and define your own environment, in which you work.

This notebook shows you how you can build your own kernel for a pyenv environment.

Attention: pyenv is a wonderful tool for managing multiple Python versions. You can easily try out new language features or help contribute to a project that is on a different version of Python. Using pyenv is also a great way to install pre-release versions of Python so that you can test them for bugs. BUT, if this is not your intention and you are fine with the default Python version of your current software stage. DO NOT use pyenv - please have a look at the general (PyPI) howto called "Create_JupyterKernel_general.ipynb" instead.

Attention: This notebook is meant to run out of a JupyterLab on JSC's HPC systems.

Building your own Jupyter kernel is a four step process¶

Install pyenv
- run the pyenv installation
Create/Pimp new virtual Python environment
- venv
Create/Edit launch script for the Jupyter kernel
- kernel.sh
Create/Edit Jupyter kernel configuration
- kernel.json

Settings¶

Set the kernel name¶

must be lower case
change if you like

In [ ]:

Copied!

# INPUT NEEDED:
KERNEL_NAME=${USER}_kernel_pyenv

export KERNEL_NAME=$(echo "${KERNEL_NAME}" | awk '{print tolower($0)}')
echo ${KERNEL_NAME} # double check
# INPUT NEEDED:
KERNEL_NAME=${USER}_kernel_pyenv

export KERNEL_NAME=$(echo "${KERNEL_NAME}" | awk '{print tolower($0)}')
echo ${KERNEL_NAME} # double check

Set the kernel directory¶

check that the kernel name is unique
print the location of the new kernel

In [ ]:

Copied!





# define KERNEL_SPECS_DIR
export KERNEL_SPECS_PREFIX=${HOME}/.local
if [ ! -d "$KERNEL_SPECS_PREFIX" ]; then
  echo "ERROR: please create directory $KERNEL_SPECS_PREFIX"
fi
export KERNEL_SPECS_DIR=${KERNEL_SPECS_PREFIX}/share/jupyter/kernels

# check if kernel name is unique
if [ -d "${KERNEL_SPECS_DIR}/${KERNEL_NAME}" ]; then
  echo "ERROR: Kernel already exists in ${KERNEL_SPECS_DIR}/${KERNEL_NAME}"
  echo "       Rename kernel name or remove directory."
fi

# print the location of the new kernel
echo ${KERNEL_SPECS_DIR}/${KERNEL_NAME} 
# define KERNEL_SPECS_DIR
export KERNEL_SPECS_PREFIX=${HOME}/.local
if [ ! -d "$KERNEL_SPECS_PREFIX" ]; then
  echo "ERROR: please create directory $KERNEL_SPECS_PREFIX"
fi
export KERNEL_SPECS_DIR=${KERNEL_SPECS_PREFIX}/share/jupyter/kernels

# check if kernel name is unique
if [ -d "${KERNEL_SPECS_DIR}/${KERNEL_NAME}" ]; then
  echo "ERROR: Kernel already exists in ${KERNEL_SPECS_DIR}/${KERNEL_NAME}"
  echo "       Rename kernel name or remove directory."
fi

# print the location of the new kernel
echo ${KERNEL_SPECS_DIR}/${KERNEL_NAME} 

Set the kernel's pyenv environment¶

by default it is located at $PROJECT
print the location of the new kernels pyenv environment

In [ ]:

Copied!





# define KERNEL_VENVS_DIR
export KERNEL_VENVS_DIR=${PROJECT}/${USER}/jupyter/kernels
mkdir -p ${KERNEL_VENVS_DIR}

# print the location of the new kernel's environment
echo ${KERNEL_VENVS_DIR}
# define KERNEL_VENVS_DIR
export KERNEL_VENVS_DIR=${PROJECT}/${USER}/jupyter/kernels
mkdir -p ${KERNEL_VENVS_DIR}

# print the location of the new kernel's environment
echo ${KERNEL_VENVS_DIR}

1. Install a new PyEnv¶

For a pyenv environment we do not use any prebuild modules of the system.
Instead we install everything from scratch with PyEnv - even Python itself.

1.1 - Get all needed data for a pyenv installation¶

Just a simple command is needed to install pyenv in $PYENV_ROOT.
For simplicity we install a unique pyenv environment per kernel.
(more)

In [ ]:

Copied!





export PYENV_ROOT=${KERNEL_VENVS_DIR}/${KERNEL_NAME}
if [ -d "${PYENV_ROOT}" ]; then
  echo "ERROR: Directory for the pyenv installation already exists: ${PYENV_ROOT}"
  echo "       Rename kernel name or remove directory."
else
  export PATH="$PYENV_ROOT/bin:$PATH"
  curl https://pyenv.run | bash
  echo ${PYENV_ROOT} # double check
fi

# print the location of the new pyenv environment
echo ${PYENV_ROOT}
export PYENV_ROOT=${KERNEL_VENVS_DIR}/${KERNEL_NAME}
if [ -d "${PYENV_ROOT}" ]; then
  echo "ERROR: Directory for the pyenv installation already exists: ${PYENV_ROOT}"
  echo "       Rename kernel name or remove directory."
else
  export PATH="$PYENV_ROOT/bin:$PATH"
  curl https://pyenv.run | bash
  echo ${PYENV_ROOT} # double check
fi

# print the location of the new pyenv environment
echo ${PYENV_ROOT}

1.2 - Initialize the pyenv installation¶

In [ ]:

Copied!





export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init --path)"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init --path)"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

1.3 - Install basic packages in the new pyenv installation¶

For these steps, make sure to have a clean & basic environment before starting this process.

In [ ]:

Copied!

module purge
module load Stages/2025
module purge
module load Stages/2025

Install a python version (e.g. 3.12.3)

Attention: You are free to change the Python version here.

In [ ]:

Copied!

export PYTHON_VER=3.12.3
pyenv install ${PYTHON_VER}
export PYTHON_VER=3.12.3
pyenv install ${PYTHON_VER}

Double check the installation

In [ ]:

Copied!





which python
which pip
python --version  # should show system python version - not the version installed above because we have not activated a virtual environment, yet.
pip list
which python
which pip
python --version  # should show system python version - not the version installed above because we have not activated a virtual environment, yet.
pip list

2. Create/Pimp new virtual Python environment¶

2.1 - Create and activate a virtual environment for the kernel¶

In [ ]:

Copied!





export PYENV_ENV=${KERNEL_NAME}
export VIRTUAL_ENV=${KERNEL_VENVS_DIR}/${KERNEL_NAME}/versions/${PYTHON_VER}/envs/${PYENV_ENV}
if [ -d "${VIRTUAL_ENV}" ]; then
  echo "ERROR: Directory for virtual environment already ${VIRTUAL_ENV}"
  echo "       Rename kernel name or remove directory."
else
  pyenv virtualenv ${PYTHON_VER} ${PYENV_ENV}
  echo ${VIRTUAL_ENV} # double check
fi
export PYENV_ENV=${KERNEL_NAME}
export VIRTUAL_ENV=${KERNEL_VENVS_DIR}/${KERNEL_NAME}/versions/${PYTHON_VER}/envs/${PYENV_ENV}
if [ -d "${VIRTUAL_ENV}" ]; then
  echo "ERROR: Directory for virtual environment already ${VIRTUAL_ENV}"
  echo "       Rename kernel name or remove directory."
else
  pyenv virtualenv ${PYTHON_VER} ${PYENV_ENV}
  echo ${VIRTUAL_ENV} # double check
fi

In [ ]:

Copied!

pyenv deactivate
pyenv activate ${PYENV_ENV}
pyenv deactivate
pyenv activate ${PYENV_ENV}

2.2 - Install whatever else you need in your Python virtual environment (using pip)¶

Jupyter requires the ipykernel module and its dependencies.

In [ ]:

Copied!

which python
python --version  # should show new installed python version (here 3.12.3)
which python
python --version  # should show new installed python version (here 3.12.3)

Attention: Now you can add extra Python packages you need.

In [ ]:

Copied!





if [ -z "${VIRTUAL_ENV}" ]; then
  echo "ERROR: Virtual environment not successfully initialized."
else
  echo "Installing custom Python packages using pip from the virtual environment:"
  which pip
  pip install ipykernel
  # pip install <python-package you need>
fi
if [ -z "${VIRTUAL_ENV}" ]; then
  echo "ERROR: Virtual environment not successfully initialized."
else
  echo "Installing custom Python packages using pip from the virtual environment:"
  which pip
  pip install ipykernel
  # pip install 
fi

3. Create/Edit launch script for the Jupyter kernel¶

3.1 - Create launch script, which loads your Python virtual environment and starts the ipykernel process inside:¶

In [ ]:

Copied!





# double check
echo ${PYENV_ENV}
echo ${PYENV_ROOT}
echo ${VIRTUAL_ENV}
# double check
echo ${PYENV_ENV}
echo ${PYENV_ROOT}
echo ${VIRTUAL_ENV}

In [ ]:

Copied!





echo '#!/bin/bash

module purge
module load Stages/2025

export PYENV_ROOT='"$PYENV_ROOT"'
export PATH='"$PYENV_ROOT"'/bin:'"$PATH"'
eval "$(pyenv init --path)"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

# Activate your Python virtual environment
pyenv activate '"${PYENV_ENV}"'

exec python -Xfrozen_modules=off -m ipykernel $@' > ${VIRTUAL_ENV}/kernel.sh
chmod +x ${VIRTUAL_ENV}/kernel.sh

echo ${VIRTUAL_ENV}/kernel.sh
cat ${VIRTUAL_ENV}/kernel.sh # double check
echo '#!/bin/bash

module purge
module load Stages/2025

export PYENV_ROOT='"$PYENV_ROOT"'
export PATH='"$PYENV_ROOT"'/bin:'"$PATH"'
eval "$(pyenv init --path)"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

# Activate your Python virtual environment
pyenv activate '"${PYENV_ENV}"'

exec python -Xfrozen_modules=off -m ipykernel $@' > ${VIRTUAL_ENV}/kernel.sh
chmod +x ${VIRTUAL_ENV}/kernel.sh

echo ${VIRTUAL_ENV}/kernel.sh
cat ${VIRTUAL_ENV}/kernel.sh # double check

4. Create/Edit Jupyter kernel configuration¶

In [ ]:

Copied!

export VIRTUAL_ENV_KERNELS=${VIRTUAL_ENV}/share/jupyter/kernels
echo ${VIRTUAL_ENV_KERNELS}
export VIRTUAL_ENV_KERNELS=${VIRTUAL_ENV}/share/jupyter/kernels
echo ${VIRTUAL_ENV_KERNELS}

4.1 - Create and adjust the kernel.json file¶

In [ ]:

Copied!





mkdir -p ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}

echo '{
  "argv": [
    "'${VIRTUAL_ENV}/kernel.sh'",
    "-m",
    "ipykernel_launcher",
    "-f",
    "{connection_file}"
  ],
  "display_name": "'${KERNEL_NAME}'",
  "language": "python",
  "metadata": {
   "debugger": true
  }
}' > ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json

echo ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json
cat ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json # double check
mkdir -p ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}

echo '{
  "argv": [
    "'${VIRTUAL_ENV}/kernel.sh'",
    "-m",
    "ipykernel_launcher",
    "-f",
    "{connection_file}"
  ],
  "display_name": "'${KERNEL_NAME}'",
  "language": "python",
  "metadata": {
   "debugger": true
  }
}' > ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json

echo ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json
cat ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json # double check

4.2 - Create link to kernel specs¶

In [ ]:

Copied!





mkdir -p ${KERNEL_SPECS_DIR}
cd ${KERNEL_SPECS_DIR}
ln -s ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME} .

echo -e "\n\nThe new kernel '${KERNEL_NAME}' was added to your kernels in '${KERNEL_SPECS_DIR}/'\n"
ls ${KERNEL_SPECS_DIR} # double check
mkdir -p ${KERNEL_SPECS_DIR}
cd ${KERNEL_SPECS_DIR}
ln -s ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME} .

echo -e "\n\nThe new kernel '${KERNEL_NAME}' was added to your kernels in '${KERNEL_SPECS_DIR}/'\n"
ls ${KERNEL_SPECS_DIR} # double check

4.3 - Use the kernel¶

You can select the new kernel in the top right corner of your notebook or from JupyterLab's Launchpad
The kernel icon will be added to your launcher, after a while by JupyterLab automatically or once you've restarted the JupyterLab