Development Workflow

This section describes the steps necessary to build Elyra in a development environment.

Requirements

Setting up your development environment

Install Miniconda Download and install a Python 3 version of Miniconda according to your Operating System
Create a new Python environment
```
conda create -n <env-name> python
```
The python version of your environment will match the miniconda version you installed. You can override the default by explicitly setting python=3.7, for example.

Activate the new environment

conda activate <env-name>

Verify your miniconda environment

python --version
which python # Displays current python path
pip3 --version
which pip3

Python path must be under miniconda envs folder. Confirm pip3 location matches where miniconda is installed.

Install NodeJS

conda install -y -c conda-forge/label/main nodejs

Setting up your Elyra Github repository

Fork the Elyra Github repository (if you haven’t already)

Make a local copy of Elyra fork

git clone https://github.com/<your-github-id>/elyra.git
cd elyra

Set upstream as described in the GitHub documentation

Building

Elyra is divided in two parts, a collection of Jupyter Notebook backend extensions, and their respective JupyterLab UI extensions. Our JupyterLab extensions are located in our packages directory.

Build & Installation

Elyra uses make to automate some of the development workflow tasks.

Issuing a make command with no task specified will provide a list of the currently supported tasks.

$ make

clean                          Make a clean source tree and uninstall extensions
container-images               Build all container images
docs                           Build docs
install-server                 Build and install backend only
install                        Build and install
lint                           Run linters
publish-container-images       Publish all container images

You can build and install all Elyra packages with:

make clean install

You can check that the notebook server extension was successfully installed with:

jupyter serverextension list

You can check that the JupyterLab extension was successfully installed with:

jupyter labextension list

Note: When switching between Elyra major versions, it is recommended to clean your JupyterLab environment before a build. The clean-jupyterlab removes your JupyterLab packages and completely deletes your Jupyter workspace. Make sure to backup any important data in your environment before running the script.

To clean your environment and install the latest JupyterLab: etc/scripts/clean-jupyterlab.sh To specify a JupyterLab version to be installed: etc/scripts/clean-jupyterlab.sh --version 2.2.9

Parallel Development with @elyra/pipeline-editor

You can install Elyra using a local build of @elyra/pipeline-editor with:

make clean dev-link install

Back-end Development

After making code changes to the back-end, you can re-build Elyra’s Python package with:

make install-server

This command builds and installs the updated Python package independently, skipping any UI component build.

Restart JupyterLab to pick up the new code changes.

Front-end Incremental Development

Elyra supports incremental development using --watch. This allows you to make code changes to front-end packages and see them without running make install again.

After installation run the following to watch for code changes and rebuild automatically:

make watch

Then in a separate terminal, using the same Python environment, start JupyterLab in watch mode:

jupyter lab --watch

When in watch mode JupyterLab will watch for changes in the build of each package and rebuild. To see your changes just refresh JupyterLab in your browser.

Note: JupyterLab watch mode will not pick up changes in package dependencies like services. So when making changes to services you will need to stop and restart jupyter lab --watch and not just refresh your browser.

Building the Elyra Container Image

Elyra’s container image can be built using:

make elyra-image

By default, the command above will build a container image from the tip of the repository master branch.

In order to build from a particular release, you can pass a TAG parameter to the make command as below:

make elyra-image TAG=2.2.1

Official container images are published on Docker Hub and quay.io.

Recipes: Configuring Apache Airflow on Kubernetes for use with Elyra

Developer guide: Conventions for contributing to Elyra Extensions