Pipeline Basics
Move data from Point A to B (or "source to target" if you want to be fancy)
To share your data with the public, we will need to move it from wherever it lives (your database, system, app, spreadsheet, etc.) onto our platform. We may also have to change columns or values before publishing it. The process of moving and cleaning data is a 'data pipeline'.
The process of moving data is often called ETL because the steps are:
- Extract data from it's source
- Perform Transformations on the data
- Load the data into it's target destination
The first question to ask yourself is, how often will this dataset update?
The next two pages cover manual and automated data pipelines.
Last modified 9mo ago