📄️ How is Starlake different ?
Looking at ELT tools, we can see that they are either:
📄️ General
Starlake is based on a YAML DSL (Domain Specific Language) to define your Extract, Load and Transform data pipelines. All these files are stored in the metadata folder.
📄️ Extract
Extraction strategies
📄️ Load
Use YAML to load data from file data sources to Starlake.
📄️ Transform
Building a datawarehouse, involves chaining complex transformations and aggregations using SQL queries.
📄️ Expectations
Starlake comes with a library of expectations that are run against your datasets at load time or after transformation. These expectations are used to validate the data and ensure that it is in the expected format.
📄️ Orchestration
Running a load or transform job is rarely a on-demand task. It is usually a scheduled job that runs periodically.
📄️ Incremental Load & Transform
Incremental Load