Author: Jens Henrik Göbbert

Index

Create your own Jupyter Kernel

Often the standard kernel do not provide all features you need for your work. This might be that certain modules are not loaded or packages are not installed.
With your own kernel you can overcome that problem easily and define your own environment, in which you work.

This notebook shows you how you can build your own kernel for a python environment.

Attention: This notebook is meant to run out of a JupyterLab on JSC's HPC systems.

Building your own Jupyter kernel is a three step process¶

Create/Pimp new virtual Python environment
- venv
Create/Edit launch script for the Jupyter kernel
- kernel.sh
Create/Edit Jupyter kernel configuration
- kernel.json

Settings¶

Set the kernel name¶

must be lower case
change if you like

In [1]:

Copied!

# INPUT NEEDED:
KERNEL_NAME=MyKernel

export KERNEL_NAME=$(echo "${KERNEL_NAME}" | awk '{print tolower($0)}')
echo ${KERNEL_NAME} # double check
# INPUT NEEDED:
KERNEL_NAME=MyKernel

export KERNEL_NAME=$(echo "${KERNEL_NAME}" | awk '{print tolower($0)}')
echo ${KERNEL_NAME} # double check

mykernel

Set the kernel directory¶

check that the kernel name is unique
print the location of the new kernel

In [3]:

Copied!





# define KERNEL_SPECS_DIR
export KERNEL_SPECS_PREFIX=${HOME}/.local
if [ ! -d "$KERNEL_SPECS_PREFIX" ]; then
  echo "ERROR: please create directory $KERNEL_SPECS_PREFIX"
fi
export KERNEL_SPECS_DIR=${KERNEL_SPECS_PREFIX}/share/jupyter/kernels

# check if kernel name is unique
if [ -d "${KERNEL_SPECS_DIR}/${KERNEL_NAME}" ]; then
  echo "ERROR: Kernel already exists in ${KERNEL_SPECS_DIR}/${KERNEL_NAME}"
  echo "       Rename kernel name or remove directory."
fi

# print the location of the new kernel
echo ${KERNEL_SPECS_DIR}/${KERNEL_NAME} 
# define KERNEL_SPECS_DIR
export KERNEL_SPECS_PREFIX=${HOME}/.local
if [ ! -d "$KERNEL_SPECS_PREFIX" ]; then
  echo "ERROR: please create directory $KERNEL_SPECS_PREFIX"
fi
export KERNEL_SPECS_DIR=${KERNEL_SPECS_PREFIX}/share/jupyter/kernels

# check if kernel name is unique
if [ -d "${KERNEL_SPECS_DIR}/${KERNEL_NAME}" ]; then
  echo "ERROR: Kernel already exists in ${KERNEL_SPECS_DIR}/${KERNEL_NAME}"
  echo "       Rename kernel name or remove directory."
fi

# print the location of the new kernel
echo ${KERNEL_SPECS_DIR}/${KERNEL_NAME} 

/home/jovyan/.local/share/jupyter/kernels/mykernel

Set the kernel's virtual environment¶

by default it is located at $PROJECT
print the location of the new kernels virtual environment

In [5]:

Copied!





# define KERNEL_VENVS_DIR
export KERNEL_VENVS_DIR=/home/jovyan/jupyter/kernels
mkdir -p ${KERNEL_VENVS_DIR}

# print the location of the new kernels virtual environment
echo ${KERNEL_VENVS_DIR}
# define KERNEL_VENVS_DIR
export KERNEL_VENVS_DIR=/home/jovyan/jupyter/kernels
mkdir -p ${KERNEL_VENVS_DIR}

# print the location of the new kernels virtual environment
echo ${KERNEL_VENVS_DIR}

/home/jovyan/jupyter/kernels

1. Create/Pimp new virtual Python environment¶

1.1 - Load basic Python module¶

In [6]:

Copied!





module purge
module load Stages/2025 # any stage can be used
module load GCC
module load Python      # only Python is mandatory
module purge
module load Stages/2025 # any stage can be used
module load GCC
module load Python      # only Python is mandatory

The following modules were not unloaded:
  (Use "module --force purge" to unload all):

  1) Stages/2025

In [7]:

Copied!

# get Python version
export PYV=$(python -c 'import sys; print(".".join(map(str, sys.version_info[:2])))')
echo $PYV
# get Python version
export PYV=$(python -c 'import sys; print(".".join(map(str, sys.version_info[:2])))')
echo $PYV

3.12

1.2 - Load extra modules you need for your kernel¶

In [ ]:

Copied!

# module load <module you need>
# module load

1.3 - Create and activate a virtual environment for the kernel¶

and ensure python packages installed in the virtual environment are always prefered

In [8]:

Copied!





export VIRTUAL_ENV=${KERNEL_VENVS_DIR}/${KERNEL_NAME}
if [ -d "${VIRTUAL_ENV}" ]; then
  echo "ERROR: Directory for virtual environment already ${VIRTUAL_ENV}"
  echo "       Rename kernel name or remove directory."
else
  python -m venv --system-site-packages ${VIRTUAL_ENV}
  source ${VIRTUAL_ENV}/bin/activate
  export PYTHONPATH=${VIRTUAL_ENV}/lib/python${PYV}/site-packages:${PYTHONPATH}
  echo ${VIRTUAL_ENV} # double check
fi
export VIRTUAL_ENV=${KERNEL_VENVS_DIR}/${KERNEL_NAME}
if [ -d "${VIRTUAL_ENV}" ]; then
  echo "ERROR: Directory for virtual environment already ${VIRTUAL_ENV}"
  echo "       Rename kernel name or remove directory."
else
  python -m venv --system-site-packages ${VIRTUAL_ENV}
  source ${VIRTUAL_ENV}/bin/activate
  export PYTHONPATH=${VIRTUAL_ENV}/lib/python${PYV}/site-packages:${PYTHONPATH}
  echo ${VIRTUAL_ENV} # double check
fi

/home/jovyan/jupyter/kernels/mykernel

1.4 - Install Python libraries required for communication with Jupyter¶

In [9]:

Copied!





which pip
if [ -z "${VIRTUAL_ENV}" ]; then
  echo "ERROR: Virtual environment not successfully initialized."
else
  pip install ipykernel
fi
which pip
if [ -z "${VIRTUAL_ENV}" ]; then
  echo "ERROR: Virtual environment not successfully initialized."
else
  pip install ipykernel
fi

~/jupyter/kernels/mykernel/bin/pip
Collecting ipykernel
  Downloading ipykernel-6.29.5-py3-none-any.whl.metadata (6.3 kB)
Collecting comm>=0.1.1 (from ipykernel)
  Downloading comm-0.2.2-py3-none-any.whl.metadata (3.7 kB)
Collecting debugpy>=1.6.5 (from ipykernel)
  Downloading debugpy-1.8.13-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.3 kB)
Collecting ipython>=7.23.1 (from ipykernel)
  Downloading ipython-9.0.2-py3-none-any.whl.metadata (4.3 kB)
Collecting jupyter-client>=6.1.12 (from ipykernel)
  Downloading jupyter_client-8.6.3-py3-none-any.whl.metadata (8.3 kB)
Collecting jupyter-core!=5.0.*,>=4.12 (from ipykernel)
  Downloading jupyter_core-5.7.2-py3-none-any.whl.metadata (3.4 kB)
Collecting matplotlib-inline>=0.1 (from ipykernel)
  Downloading matplotlib_inline-0.1.7-py3-none-any.whl.metadata (3.9 kB)
Collecting nest-asyncio (from ipykernel)
  Downloading nest_asyncio-1.6.0-py3-none-any.whl.metadata (2.8 kB)
Requirement already satisfied: packaging in /p/software/jsccloud/stages/2025/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/site-packages (from ipykernel) (24.0)
Collecting psutil (from ipykernel)
  Downloading psutil-7.0.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (22 kB)
Collecting pyzmq>=24 (from ipykernel)
  Downloading pyzmq-26.3.0-cp312-cp312-manylinux_2_28_x86_64.whl.metadata (6.2 kB)
Collecting tornado>=6.1 (from ipykernel)
  Downloading tornado-6.4.2-cp38-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.5 kB)
Collecting traitlets>=5.4.0 (from ipykernel)
  Downloading traitlets-5.14.3-py3-none-any.whl.metadata (10 kB)
Collecting decorator (from ipython>=7.23.1->ipykernel)
  Downloading decorator-5.2.1-py3-none-any.whl.metadata (3.9 kB)
Collecting ipython-pygments-lexers (from ipython>=7.23.1->ipykernel)
  Downloading ipython_pygments_lexers-1.1.1-py3-none-any.whl.metadata (1.1 kB)
Collecting jedi>=0.16 (from ipython>=7.23.1->ipykernel)
  Downloading jedi-0.19.2-py2.py3-none-any.whl.metadata (22 kB)
Collecting pexpect>4.3 (from ipython>=7.23.1->ipykernel)
  Downloading pexpect-4.9.0-py2.py3-none-any.whl.metadata (2.5 kB)
Collecting prompt_toolkit<3.1.0,>=3.0.41 (from ipython>=7.23.1->ipykernel)
  Downloading prompt_toolkit-3.0.50-py3-none-any.whl.metadata (6.6 kB)
Collecting pygments>=2.4.0 (from ipython>=7.23.1->ipykernel)
  Downloading pygments-2.19.1-py3-none-any.whl.metadata (2.5 kB)
Collecting stack_data (from ipython>=7.23.1->ipykernel)
  Downloading stack_data-0.6.3-py3-none-any.whl.metadata (18 kB)
Collecting python-dateutil>=2.8.2 (from jupyter-client>=6.1.12->ipykernel)
  Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting platformdirs>=2.5 (from jupyter-core!=5.0.*,>=4.12->ipykernel)
  Downloading platformdirs-4.3.7-py3-none-any.whl.metadata (11 kB)
Collecting parso<0.9.0,>=0.8.4 (from jedi>=0.16->ipython>=7.23.1->ipykernel)
  Downloading parso-0.8.4-py2.py3-none-any.whl.metadata (7.7 kB)
Collecting ptyprocess>=0.5 (from pexpect>4.3->ipython>=7.23.1->ipykernel)
  Downloading ptyprocess-0.7.0-py2.py3-none-any.whl.metadata (1.3 kB)
Collecting wcwidth (from prompt_toolkit<3.1.0,>=3.0.41->ipython>=7.23.1->ipykernel)
  Downloading wcwidth-0.2.13-py2.py3-none-any.whl.metadata (14 kB)
Collecting six>=1.5 (from python-dateutil>=2.8.2->jupyter-client>=6.1.12->ipykernel)
  Downloading six-1.17.0-py2.py3-none-any.whl.metadata (1.7 kB)
Collecting executing>=1.2.0 (from stack_data->ipython>=7.23.1->ipykernel)
  Downloading executing-2.2.0-py2.py3-none-any.whl.metadata (8.9 kB)
Collecting asttokens>=2.1.0 (from stack_data->ipython>=7.23.1->ipykernel)
  Downloading asttokens-3.0.0-py3-none-any.whl.metadata (4.7 kB)
Collecting pure-eval (from stack_data->ipython>=7.23.1->ipykernel)
  Downloading pure_eval-0.2.3-py3-none-any.whl.metadata (6.3 kB)
Downloading ipykernel-6.29.5-py3-none-any.whl (117 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 117.2/117.2 kB 4.2 MB/s eta 0:00:00
Downloading comm-0.2.2-py3-none-any.whl (7.2 kB)
Downloading debugpy-1.8.13-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.2/4.2 MB 19.7 MB/s eta 0:00:00:00:01
Downloading ipython-9.0.2-py3-none-any.whl (600 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 600.5/600.5 kB 3.5 MB/s eta 0:00:00:00:01
Downloading jupyter_client-8.6.3-py3-none-any.whl (106 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 106.1/106.1 kB 4.7 MB/s eta 0:00:00
Downloading jupyter_core-5.7.2-py3-none-any.whl (28 kB)
Downloading matplotlib_inline-0.1.7-py3-none-any.whl (9.9 kB)
Downloading pyzmq-26.3.0-cp312-cp312-manylinux_2_28_x86_64.whl (859 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 859.9/859.9 kB 6.6 MB/s eta 0:00:00:00:01
Downloading tornado-6.4.2-cp38-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (437 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 437.2/437.2 kB 7.1 MB/s eta 0:00:00:00:01
Downloading traitlets-5.14.3-py3-none-any.whl (85 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 85.4/85.4 kB 3.5 MB/s eta 0:00:00
Downloading nest_asyncio-1.6.0-py3-none-any.whl (5.2 kB)
Downloading psutil-7.0.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (277 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 278.0/278.0 kB 4.0 MB/s eta 0:00:00:00:01
Downloading jedi-0.19.2-py2.py3-none-any.whl (1.6 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 18.4 MB/s eta 0:00:00:00:01
Downloading pexpect-4.9.0-py2.py3-none-any.whl (63 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 63.8/63.8 kB 2.3 MB/s eta 0:00:00
Downloading platformdirs-4.3.7-py3-none-any.whl (18 kB)
Downloading prompt_toolkit-3.0.50-py3-none-any.whl (387 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 387.8/387.8 kB 4.2 MB/s eta 0:00:00:00:01
Downloading pygments-2.19.1-py3-none-any.whl (1.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 9.4 MB/s eta 0:00:000:00:01
Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 229.9/229.9 kB 2.5 MB/s eta 0:00:00:00:01
Downloading decorator-5.2.1-py3-none-any.whl (9.2 kB)
Downloading ipython_pygments_lexers-1.1.1-py3-none-any.whl (8.1 kB)
Downloading stack_data-0.6.3-py3-none-any.whl (24 kB)
Downloading asttokens-3.0.0-py3-none-any.whl (26 kB)
Downloading executing-2.2.0-py2.py3-none-any.whl (26 kB)
Downloading parso-0.8.4-py2.py3-none-any.whl (103 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 103.7/103.7 kB 4.1 MB/s eta 0:00:00
Downloading ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB)
Downloading six-1.17.0-py2.py3-none-any.whl (11 kB)
Downloading pure_eval-0.2.3-py3-none-any.whl (11 kB)
Downloading wcwidth-0.2.13-py2.py3-none-any.whl (34 kB)
Installing collected packages: wcwidth, pure-eval, ptyprocess, traitlets, tornado, six, pyzmq, pygments, psutil, prompt_toolkit, platformdirs, pexpect, parso, nest-asyncio, executing, decorator, debugpy, asttokens, stack_data, python-dateutil, matplotlib-inline, jupyter-core, jedi, ipython-pygments-lexers, comm, jupyter-client, ipython, ipykernel
Successfully installed asttokens-3.0.0 comm-0.2.2 debugpy-1.8.13 decorator-5.2.1 executing-2.2.0 ipykernel-6.29.5 ipython-9.0.2 ipython-pygments-lexers-1.1.1 jedi-0.19.2 jupyter-client-8.6.3 jupyter-core-5.7.2 matplotlib-inline-0.1.7 nest-asyncio-1.6.0 parso-0.8.4 pexpect-4.9.0 platformdirs-4.3.7 prompt_toolkit-3.0.50 psutil-7.0.0 ptyprocess-0.7.0 pure-eval-0.2.3 pygments-2.19.1 python-dateutil-2.9.0.post0 pyzmq-26.3.0 six-1.17.0 stack_data-0.6.3 tornado-6.4.2 traitlets-5.14.3 wcwidth-0.2.13

[notice] A new release of pip is available: 24.0 -> 25.0.1
[notice] To update, run: pip install --upgrade pip

1.5 - Install whatever else you need in your Python virtual environment (using pip)¶

In [ ]:

Copied!

#pip install <python-package you need>
#pip install

2. Create/Edit launch script for the Jupyter kernel¶

2.1 - Create launch script, which loads your Python virtual environment and starts the ipykernel process inside:¶

Attention: You MUST load the exactly the same modules as you did above for your virtual Python environment.

In [10]:

Copied!





echo '#!/bin/bash'"

# Load basic Python module
module purge
module load Stages/2025
module load GCC
module load Python

# Load extra modules you need for your kernel (as you did in step 1.2)
#module load <module you need>

# Activate your Python virtual environment
source ${VIRTUAL_ENV}/bin/activate
    
# Ensure python packages installed in the virtual environment are always prefered
export PYTHONPATH=${VIRTUAL_ENV}/lib/python${PYV}/site-packages:"'${PYTHONPATH}'"
    
exec python -m ipykernel "'$@' > ${VIRTUAL_ENV}/kernel.sh
chmod +x ${VIRTUAL_ENV}/kernel.sh

cat ${VIRTUAL_ENV}/kernel.sh # double check
echo '#!/bin/bash'"

# Load basic Python module
module purge
module load Stages/2025
module load GCC
module load Python

# Load extra modules you need for your kernel (as you did in step 1.2)
#module load 

# Activate your Python virtual environment
source ${VIRTUAL_ENV}/bin/activate
    
# Ensure python packages installed in the virtual environment are always prefered
export PYTHONPATH=${VIRTUAL_ENV}/lib/python${PYV}/site-packages:"'${PYTHONPATH}'"
    
exec python -m ipykernel "'$@' > ${VIRTUAL_ENV}/kernel.sh
chmod +x ${VIRTUAL_ENV}/kernel.sh

cat ${VIRTUAL_ENV}/kernel.sh # double check

#!/bin/bash

# Load basic Python module
module purge
module load Stages/2025
module load GCC
module load Python

# Load extra modules you need for your kernel (as you did in step 1.2)
#module load <module you need>

# Activate your Python virtual environment
source /home/jovyan/jupyter/kernels/mykernel/bin/activate
    
# Ensure python packages installed in the virtual environment are always prefered
export PYTHONPATH=/home/jovyan/jupyter/kernels/mykernel/lib/python3.12/site-packages:${PYTHONPATH}
    
exec python -m ipykernel $@

3. Create/Edit Jupyter kernel configuration¶

3.1 - Create Jupyter kernel configuration directory and files¶

In [11]:

Copied!

python -m ipykernel install --name=${KERNEL_NAME} --prefix ${VIRTUAL_ENV}
export VIRTUAL_ENV_KERNELS=${VIRTUAL_ENV}/share/jupyter/kernels
python -m ipykernel install --name=${KERNEL_NAME} --prefix ${VIRTUAL_ENV}
export VIRTUAL_ENV_KERNELS=${VIRTUAL_ENV}/share/jupyter/kernels

Installed kernelspec mykernel in /home/jovyan/jupyter/kernels/mykernel/share/jupyter/kernels/mykernel

3.2 - Adjust kernel.json file¶

In [12]:

Copied!





mv ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json.orig

echo '{
  "argv": [
    "'${KERNEL_VENVS_DIR}/${KERNEL_NAME}/kernel.sh'",
    "-m",
    "ipykernel_launcher",
    "-f",
    "{connection_file}"
  ],
  "display_name": "'${KERNEL_NAME}'",
  "language": "python",
  "metadata": {
   "debugger": true
  }
}' > ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json

cat ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json # double check
mv ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json.orig

echo '{
  "argv": [
    "'${KERNEL_VENVS_DIR}/${KERNEL_NAME}/kernel.sh'",
    "-m",
    "ipykernel_launcher",
    "-f",
    "{connection_file}"
  ],
  "display_name": "'${KERNEL_NAME}'",
  "language": "python",
  "metadata": {
   "debugger": true
  }
}' > ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json

cat ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME}/kernel.json # double check

{
  "argv": [
    "/home/jovyan/jupyter/kernels/mykernel/kernel.sh",
    "-m",
    "ipykernel_launcher",
    "-f",
    "{connection_file}"
  ],
  "display_name": "mykernel",
  "language": "python",
  "metadata": {
   "debugger": true
  }
}

3.3 - Create link to kernel specs¶

In [13]:

Copied!





mkdir -p ${KERNEL_SPECS_DIR}
cd ${KERNEL_SPECS_DIR}
ln -s ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME} .

echo -e "\n\nThe new kernel '${KERNEL_NAME}' was added to your kernels in '${KERNEL_SPECS_DIR}/'\n"
ls ${KERNEL_SPECS_DIR} # double check
mkdir -p ${KERNEL_SPECS_DIR}
cd ${KERNEL_SPECS_DIR}
ln -s ${VIRTUAL_ENV_KERNELS}/${KERNEL_NAME} .

echo -e "\n\nThe new kernel '${KERNEL_NAME}' was added to your kernels in '${KERNEL_SPECS_DIR}/'\n"
ls ${KERNEL_SPECS_DIR} # double check


The new kernel 'mykernel' was added to your kernels in '/home/jovyan/.local/share/jupyter/kernels/'

abc  mykernel

3.4 - Use the kernel¶

You can select the new kernel in the top right corner of your notebook or from JupyterLab's Launchpad
The kernel icon will be added to your launcher, after a while by JupyterLab automatically or once you've restarted the JupyterLab

4. Cleanup¶

In [14]:

Copied!

deactivate
deactivate

In [ ]: