Search…
JupyterLab Installation and Configuration
You can use JupyterLab to access OmniSciDB.

Installing Jupyter with OmniSciDB

When OmniSci is running in Docker, the paths used must be relative paths accessible to Docker, instead of OS-level paths. Keep the following in mind when performing Docker-based installations:
    You must locate your docker-compose.yml file in a location that is reachable from Docker.
    The ingest and export paths from OmniSci are likely different from the actual location of the file because OmniSci uses container paths instead of operating system paths.
    1.
    Install the NVIDIA drivers and nvidia-container-runtime for your operating system, using the instructions at https://github.com/NVIDIA/nvidia-container-runtime.
    2.
    For Apt-based installations, such as Ubuntu, use the Docker preparation instructions.
    3.
    Change the default Docker runtime to nvidia and restart Docker.
    a. Edit /etc/docker/daemon.jsonand add "default-runtime": "nvidia". The resulting file should look similar to the following:
    1
    {
    2
    "default-runtime": "nvidia",
    3
    "runtimes": {
    4
    "nvidia": {
    5
    "path": "nvidia-container-runtime",
    6
    "runtimeArgs": []
    7
    }
    8
    }
    9
    }
    Copied!
    b. Restart the Docker service:
    1
    sudo systemctl restart docker
    Copied!
    c. Validate NVIDIA Docker:
    1
    docker run --rm nvidia/cudagl:10.1-runtime-ubuntu18.04 nvidia-smi
    Copied!
    4.
    Install docker-compose version 1.25.1 or higher:
    1
    sudo curl -L https://github.com/docker/compose/releases/download/1.25.1/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
    2
    sudo chmod +x /usr/local/bin/docker-compose
    Copied!
    5.
    Create an OmniSci storage directory with a name of your choosing and change directories to it.
    1
    sudo mkdir /var/lib/omnisci
    2
    cd /var/lib/omnisci
    Copied!
    6.
    Create the file /var/lib/omnisci/omnisci.conf, and configure the jupyter-url setting under the [web] section to point to the Jupyter service:
    1
    [web]
    2
    jupyter-url = "http://jupyterhub:8000"
    3
    servers-json = "/omnisci-storage/servers.json"
    Copied!
    7.
    Create the file /var/lib/omnisci/data/omnisci.license. Copy your license key from the registration email message. If you have not received your license key, contact your Sales Representative or register for your 30-day trial here.
    8.
    Create the following /var/lib/omnisci/servers.json entry to enable Jupyter features in Immerse.
    1
    [
    2
    {
    3
    "enableJupyter": true
    4
    }
    5
    ]
    Copied!
    9.
    Create /var/lib/omnisci/docker-compose.yml:
    1
    version: '3.7'
    2
    3
    services:
    4
    5
    omnisciserver:
    6
    container_name: omnisciserver
    7
    image: omnisci/omnisci-ee-cuda:v5.5.6
    8
    restart: always
    9
    ipc: shareable
    10
    volumes:
    11
    - /var/lib/omnisci:/omnisci-storage
    12
    - /var/lib/omnisci/data/omnisci_import/jhub_omnisci_dropbox:/jhub_omnisci_dropbox
    13
    networks:
    14
    - omnisci-frontend
    15
    - omnisci-backend
    16
    ports:
    17
    - "6273:6273"
    18
    - "6274:6274"
    19
    - "6278:6278"
    20
    # If using binary encryption, uncomment the below line to override the default
    21
    # command that uses startomnisci, noting that you must have an existing OmniSci
    22
    # data directory and run initdb before making this change.
    23
    # command: /bin/bash -c "/omnisci/bin/omnisci_web_server --config /omnisci-storage/omnisci.conf & /omnisci/bin/omnisci_server --config /omnisci-storage/omnisci.conf"
    24
    25
    26
    jupyterhub:
    27
    container_name: jupyterhub
    28
    image: omnisci/omnisci-jupyterhub:v0.4
    29
    restart: always
    30
    networks:
    31
    - omnisci-backend
    32
    depends_on:
    33
    - omnisciserver
    34
    35
    volumes:
    36
    - /var/run/docker.sock:/var/run/docker.sock
    37
    38
    # Map this volume if binary encryption mode is configured and certificates are being validated
    39
    # - /var/lib/omnisci/cacerts.crt:/omnisci-storage/cacerts.crt
    40
    41
    environment:
    42
    ### Required settings ###
    43
    44
    # Note: if changing the image, do not remove the &jlabimage anchor
    45
    # The gpu image includes Rapids, requires GPU devices and nvidia docker runtime:
    46
    # DOCKER_JUPYTER_LAB_IMAGE: &jlabimage omnisci/omnisci-jupyterlab-gpu:v0.3
    47
    # Without rapids does not require GPU:
    48
    DOCKER_JUPYTER_LAB_IMAGE: &jlabimage omnisci/omnisci-jupyterlab-cpu:v0.4
    49
    50
    OMNISCI_HOST: omnisciserver
    51
    OMNISCI_JUPYTER_ROLE: omnisci_jupyter
    52
    53
    ### Optional settings ###
    54
    55
    # DOCKER_NOTEBOOK_DIR: /home/jovyan # The directory inside the user's Jupyter Lab container to mount the user volume to.
    56
    # HUB_IP: jupyterhub # The hostname or IP of the Jupyter Hub server
    57
    # JHUB_BASE_URL: /jupyter/ # The base URL prepended to all Jupyter Hub and Lab requests
    58
    # JHUB_ENABLE_ADMIN_ACCESS: "false" # Enable / disable admin access to Jupyter Hub
    59
    # JLAB_CONTAINER_AUTOREMOVE: "true" # Enable / disable automatic removal of stopped Juptyer Lab containers
    60
    # JLAB_DOCKER_NETWORK_NAME: omnisci-backend # The docker network name for Jupyter Lab containers
    61
    # JLAB_IDLE_TIMEOUT: 3600 # Shut down Jupyter Lab containers after this many seconds of idle time
    62
    # JLAB_NAME_PREFIX: jupyterlab # Container name prefix for Lab containers
    63
    # JLAB_OMNISCI_IMPORT_VOLUME_PATH: /var/lib/omnisci/data/omnisci_import/jhub_omnisci_dropbox # Local Docker host path for where to mount the shared directory available to the OmniSciDB server for file ingest
    64
    # JLAB_NOTEBOOK_TERMINALS_ENBLED: "false" # Enable terminals in notebooks
    65
    # JLAB_USER_VOLUME_PATH: /var/lib/omnisci/jupyter/ # Local Docker host path to be used for user Jupyter Lab home directory volume mapping
    66
    # JUPYTER_DEBUG: "true" # Turn on / off debugging for Jupyter Hub and Lab
    67
    # OMNISCI_ALLOW_SUPERUSER_ROLE: "false" # Authorize super users access to Jupyter without needing to be given the OmniSci role specified in OMNISCI_JUPYTER_ROLE
    68
    # OMNISCI_BINARY_TLS_CACERTS_LOCAL_PATH: /var/lib/omnisci/cacerts.crt # Specifying this or mapping a volume in the Hub container to /omnisci-storage/cacerts.crt will automatically enable binary TLS mode
    69
    # OMNISCI_BINARY_TLS_VALIDATE: "false" # Whether or not to validate certificates in binary TLS mode. Specifying either "true" or "false" will enable binary TLS mode
    70
    # OMNISCI_PORT: 6278 # Port that Jupyter Hub should use to connect to OmniSci. Ensure this matches the protocol
    71
    # OMNISCI_PROTOCOL: http # Protocol that Jupyter Hub should use to connect to OmniSci. Ensure this is "binary" if using binary encryption
    72
    # SPAWNER_CPU_LIMIT: 1 # Number of CPU cores available for each Jupyter Lab container, see https://jupyterhub.readthedocs.io/en/stable/api/spawner.html#jupyterhub.spawner.Spawner.cpu_limit
    73
    # SPAWNER_RAM_LIMIT: 10G # Amount of CPU RAM available for each Jupyter Lab container, see https://jupyterhub.readthedocs.io/en/stable/api/spawner.html#jupyterhub.spawner.LocalProcessSpawner.mem_limit
    74
    # SPAWNER_ENV_OMNISCI_HOST: omnisciserver # Hostname / IP address of the OmniSci server for Lab containers to connect to by default
    75
    # SPAWNER_ENV_OMNISCI_PORT: 6274 # Port of the OmniSci server for Lab containers to connect to by default
    76
    # SPAWNER_ENV_OMNISCI_PROTOCOL: binary # Protocol of the OmniSci server for Lab containers to connect to by default
    77
    # OMNISCI_DB_URL: "omnisci://omnisciserver:6274/omnisci" # Alternative, direct connection (not Immerse session) to omnisciserver. Username will be inferred, but password will be required in notebook.
    78
    79
    # More volumes for lab containers:
    80
    # JLAB_VOLUME_1: /data1:/data1:rw
    81
    # JLAB_VOLUME_2: /var/lib/omnisci:/omnisci-storage:ro
    82
    83
    84
    # The purpose of this is to make sure the jlabimage is pulled because jupyterhub will not pull it automatically when launching
    85
    jupyterlab-tmp:
    86
    image: *jlabimage
    87
    command: echo
    88
    networks:
    89
    - omnisci-backend
    90
    91
    92
    networks:
    93
    omnisci-frontend:
    94
    driver: bridge
    95
    name: omnisci-frontend
    96
    omnisci-backend:
    97
    driver: bridge
    98
    name: omnisci-backend
    Copied!
    10.
    Run docker-compose pull to download the images from Docker Hub. Optional when you specify a version in docker-compose.yml.
    11.
    Make sure you are in the directory that you created in step 5, and run compose in detached mode:
    1
    docker-compose up -d
    Copied!
    compose restarts by default whenever stopped.
    12.
    Log in as the super user (admin/HyperInteractive).
    13.
    Create required users in OmniSciDB.
    14.
    Create the omnisci_jupyter role in OmniSciDB:
    1
    CREATE ROLE omnisci_jupyter
    Copied!
    15.
    Grant the omnisci_jupyter role to users who require Jupyter access:
    1
    GRANT omnisci_jupyter TO username
    Copied!
You might need to stop lab containers for them to be restarted with the new image:
1
docker ps | grep jupyterlab
2
docker rm -f jupyterlab-USERNAME
Copied!
You should now see Jupyter icons in the upper right of Immerse and when running queries in SQL Editor.

Adding Jupyter to an Existing OmniSciDB Instance

When OmniSci is running in Docker, the paths used must be relative paths accessible to Docker, intead of OS-level paths. Keep the following in mind when peforming Docker-based installations:
    You must locate your docker-compose.yml file in a location that is reachable from Docker.
    The ingest and export paths from OmniSci are likely different from the actual location of the file because OmniSci uses container paths instead of operating system paths.
To use Jupyter with an existing, non-Docker install of OmniSci, change OmniSci to run on Docker instead of the host. The steps are the same as the install instructions, with the following exceptions:
    1.
    Change the volume mappings to point to your existing installation path:
    1
    omnisciserver:
    2
    container_name: omnisciserver
    3
    image: omnisci/omnisci-ee-cuda:v5.5.6
    4
    restart: always
    5
    ipc: shareable
    6
    volumes:
    7
    - /var/lib/omnisci:/omnisci-storage
    8
    - /var/lib/omnisci/data/omnisci_import/jhub_omnisci_dropbox:/jhub_omnisci_dropbox
    Copied!
    2.
    Enable the following environment variables and change the relevant paths to your existing installation:
    1
    ### Optional settings ###
    2
    # DOCKER_NOTEBOOK_DIR: /home/jovyan # The directory inside the user's Jupyter Lab container to mount the user volume to.
    3
    # HUB_IP: jupyterhub # The hostname or IP of the Jupyter Hub server
    4
    # JHUB_BASE_URL: /jupyter/ # The base URL prepended to all Jupyter Hub and Lab requests
    5
    # JHUB_ENABLE_ADMIN_ACCESS: "false" # Enable / disable admin access to Jupyter Hub
    6
    # JLAB_DOCKER_NETWORK_NAME: omnisci-backend # The docker network name for Jupyter Lab containers
    7
    # JLAB_IDLE_TIMEOUT: 3600 # Shut down Jupyter Lab containers after this many seconds of idle time
    8
    # JLAB_NAME_PREFIX: jupyterlab # Container name prefix for Lab containers
    9
    JLAB_OMNISCI_IMPORT_VOLUME_PATH: /var/lib/omnisci/data/omnisci_import/jhub_omnisci_dropbox # Local Docker host path for where to mount the shared directory available to the OmniSciDB server for file ingest
    10
    # JLAB_NOTEBOOK_TERMINALS_ENBLED: "false" # Enable terminals in notebooks
    11
    JLAB_USER_VOLUME_PATH: /var/lib/omnisci/jupyter/ # Local Docker host path to be used for user Jupyter Lab home directory volume mapping
    12
    # JUPYTER_DEBUG: "true" # Turn on / off debugging for Jupyter Hub and Lab
    13
    # OMNISCIDB_CONTAINER_NAME: omnisciserver # OmniSciDB container name for IPC sharing with Lab containers
    14
    # SPAWNER_CPU_LIMIT: 1 # Number of CPU cores available for each Jupyter Lab container, see https://jupyterhub.readthedocs.io/en/stable/api/spawner.html#jupyterhub.spawner.Spawner.cpu_limit
    15
    # SPAWNER_RAM_LIMIT: 10G # Amount of CPU RAM available for each Jupyter Lab container, see https://jupyterhub.readthedocs.io/en/stable/api/spawner.html#jupyterhub.spawner.LocalProcessSpawner.mem_limit
    16
    # SPAWNER_ENV_OMNISCI_HOST: omnisciserver # Hostname / IP address of the OmniSci server for Lab containers to connect to by default
    17
    # SPAWNER_ENV_OMNISCI_PORT: 6274 # Port of the OmniSci server for Lab containers to connect to by default
    18
    # SPAWNER_ENV_OMNISCI_PROTOCOL: binary # Protocol of the OmniSci server for Lab containers to connect to by default
    Copied!
    3.
    If you have an existing omnisci.conf file:
      Add the required sections instead of creating a new file:
      1
      [web]
      2
      jupyter-url = "http://jupyterhub:8000"
      3
      servers-json = "/omnisci-storage/servers.json"
      Copied!
      Remove the data, port, http-port, and frontend properties, which should not be changed with a Docker installation.
      Ensure that all paths, such as cert and key, are accessible by Docker.
    4.
    If you have an existing servers.json file, move it to your OmniSci home directory (/var/lib/omnisci by default) and add the following key/value pair:
    1
    "enableJupyter": true
    Copied!
Before running docker-compose up -d, ensure that any existing installations are stopped and disabled. For example:
1
sudo systemctl stop omnisci_server
2
sudo systemctl stop omnisci_web_server
3
sudo systemctl disable omnisci_server
4
sudo systemctl disable omnisci_web_server
Copied!
As with any software upgrade, back up your data before you upgrade OmniSci.

Creating the jhub_omnisci_dropbox Directory

Run the following commands to create the jhub_omnisci_dropbox directory and make it writeable by your users. Change the volume mappings to point to your existing installation path:
1
sudo mkdir /var/lib/omnisci/data/omnisci_import/jhub_omnisci_dropbox
2
sudo chown 1000 /var/lib/omnisci/data/omnisci_import/jhub_omnisci_dropbox
3
sudo chmod 750 /var/lib/omnisci/data/omnisci_import/jhub_omnisci_dropbox
Copied!
This allows Jupyter users to write files into /home/jovyan/jhub_omnisci_dropbox/. You can also use that directory path in COPY FROM SQL commands.

Upgrading docker-compose Services

To upgrade Jupyter images using the docker-compose.yml file, edit docker-compose.yml as follows:
1
services:
2
omnisciserver:
3
...
4
image: omnisci/omnisci-ee-cuda:v5.5.6
5
...
6
jupyterhub:
7
container_name: jupyterhub
8
image: omnisci/omnisci-jupyterhub:v0.4
9
...
10
environment:
11
...
12
DOCKER_JUPYTER_LAB_IMAGE: &jlabimage omnisci/omnisci-jupyterlab-gpu:v0.3
Copied!
Then, use the following commands to download the images and restart the services with the new versions:
1
docker-compose pull
2
docker-compose up -d
Copied!
You might also need to stop lab containers in to start them again in a new image:
1
docker ps | grep jupyterlab
Copied!
For each user, run the following command:
1
docker rm -f jupyterlab-<username>
Copied!

Using Jupyter

Open JupyterLab by clicking the Jupyter icon in the upper right corner of Immerse.
JupyterLab opens in a new tab. You are signed in automatically using OmniSci authentication, with a notebook ready to start an OmniSci connection. The notebook is saved for you at the root of your Jupyter file system.
You can verify the location of your file by clicking the folder icon in the top left.
The contents of the cell are prefilled with explanatory comments and the omnisci_connect() method ready to set up your Omnisci connection in Jupyter. Click the Play button to run the connection statement and list the tables in your OmniSciDB instance.
You can continue to use con to run more Ibis expressions in other cells.
The connection reuses the session already in use by Immerse by passing Jupyter the raw session ID. If you connect this way without credentials, the connection has a time-to-live (TTL). After the session timeout period passes with no activity (60 minutes by default), the session is invalidated. You have to reenter Jupyter from Immerse in the same way to reestablish the connection, or use the pymapd connect method to enter your credentials manually.
You can also launch Jupyter from the Immerse SQL Editor. After you run a query in the SQL Editor, you see a button that allows you to send your query to Jupyter.
The query displays in a different notebook, ready to run the query. You must run the cell yourself to send the query and see the results.
Last modified 6mo ago