Construction
The Pipelines page can be accessed from the left-hand sidebar by clicking on Pipelines
Assembling the pipeline
Pipelines are assembled via the Graph tab. Here, pipeline tasks are represented as nodes on a DAG.
The available bank of tasks for a given pipeline are shown in the list on the bottom right (for adding to this bank, see Task Components). Click Add next to each Task to add it to the Pipeline Builder. Click and drag to move the tasks around.
Tasks can be connected together by dragging between the dots from one Task to another. The Pipeline Builder will automatically perform checks to ensure that any connection made is valid, e.g., no cycles on save.
To save process in the Pipeline builder, click the “Save” button on the (top-left in the graph).
Warning
Navigating away from a given pipeline in the Pipeline Builder view will clear any unsaved progress. The Kaspian team recommends saving progress regularly to prevent work from being lost.Versioning
Pipeline snapshots are static, permanently fixed versions of a given pipeline. To preserve code integrity during a pipeline run, only snapshots are actually executable. This also means that only the Root version of the pipeline is modifiable in the Pipeline Builder.
Press the Snapshot button in the Pipeline Builder toolbar to take a snapshot. These snapshots can then be deployed at the specified schedule from the Overview tab.
Duplicating Pipelines
Pressing the Duplicate button in the Pipeline Builder toolbar will make a copy of a pipeline (and all its tasks and schemas) at the selected version in the Pipeline Builder. Since duplicates created from any version of a pipeline will always create a new Root pipeline, duplicates can be used for branched development off of specific pipeline versions.