Installation

Starlake CLI

Prerequisites

Make sure you have Java 11+ installed on your machine.

You can check your Java version by typing java -version in a terminal.

If you don't have Java 11+ installed, you can download it from Oracle JDK or OpenJDK

Install Starlake

To install starlake, you need to download the setup script from github. The script will in turn download required dependencies and copy them to the bin subdirectory.

Linux/MacOS

sh <(curl https://raw.githubusercontent.com/starlake-ai/starlake/master/distrib/setup.sh)

Windows

Invoke-Expression (Invoke-WebRequest -Uri "https://raw.githubusercontent.com/starlake-ai/starlake/master/distrib/setup.ps1").Content

Docker

Pull the docker image from the docker hub

$ docker pull starlakeai/starlake:1.2.0

You may also run Starlake from a docker container. To do so, download this Dockerfile and build your docker image

$ git clone git@github.com:starlake-ai/starlake.git
$ cd starlake
$ docker build -t starlakeai/starlake  .
$ docker run -it starlake help

To build the docker image with a specific version of Starlake or a specific branch, you can use the following command:

$ docker build -t starlakeai/starlake:1.2.0 --build-arg SL_VERSION=1.2.0 .

To execute the docker image, you can use the following command:

$ docker run -it -v /path/to/starlake/project:/starlake starlake <command>

Note that you can pass environment variables to the docker image to configure the CLI. For example, to run against AWS redshift, you can pass the following environment variables:

$ SL_ROOT="s3a://my-bucket/my-starlake-project-base-dir/"
$ docker run -e SL_ROOT=$SL_ROOT \
    -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
    -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
    -e AWS_SESSION_TOKEN=$AWS_SESSION_TOKEN \
    -e REDSHIFT_PASSWORD=$REDSHIFT_PASSWORD \
    -it starlake <command>

The following folders should now have been created and contain Starlake dependencies.

starlake
└── bin
    ├── deps
    ├── sl
    └── spark

note

Any extra library you may need (Oracle client for example) need to be copied in the bin/deps folder.

Starlake is now installed with all its dependencies. You can run the CLI by typing starlake.

This will display the commands supported by the CLI.

Starlake Version 1.2.0
Usage:
    starlake [command]
Available commands =>
    lineage
    bootstrap
    bq2yml or bq-info
    compare
    cnxload
    esload
    extract-data
    extract-schema
    import
    infer-schema
    kafkaload
    load
    metrics
    parquet2csv
    transform
    watch
    xls2yml
    yml2ddl
    table-dependencies
    yml2xls

That's it! We now need to bootstrap a new project.

Graph Visualization

Starlake provides features to visualize the lineage of a table, the relationship between tables, and table level and row level acess policies.

To use these features, you need to install the GraphViz on top of which the starlake graph generator is built.

Linux

sudo [apt|yum] install graphviz

MacOS

brew install graphviz

Windows

Download and install one of the packages from GraphViz.

Docker

GraphViz comes pre-installed with the starlake docker image.

VS Code extension

Starlake comes with a vs-code plugin that allows you to interact with the Starlake CLI. You can install it from the vs-code marketplace.