Climate Tutorial: a Bronze–Silver–Gold Pipeline on Open Data

This tutorial builds a complete Dorieh pipeline — and then generates its documentation — using only openly available data, so anyone can reproduce every step without a data use agreement. Starting from gridded daily temperature (gridMET), the pipeline aggregates the grid over ZIP Code Tabulation Areas, loads the result into PostgreSQL, and derives Bronze, Silver and Gold layers with the Dorieh data-modeling DSL. The same design patterns apply directly to health and claims data; they are applied to a production-scale case in the Medicare claims pipeline tutorial.

The tutorial has three parts, meant to be followed in order:

• Part 1. Building the pipeline
  • Introduction
  • Prerequisites
  • Design overview
  • Directory layout
  • Step 1. Create a minimal CWL workflow skeleton
  • Step 2. Iteratively Defining Steps and Parameters
  • Step 3. Parameterize for a single day (“toy” run)
  • Step 4. Add database integration (PostgreSQL)
  • Step 5. Building Medallion Layers (Bronze, Silver, Gold)
  • Step 6. Testing the Pipeline
  • Next Steps
• Part 2. Documenting the workflow
  • Workflow files
  • Generating skeleton documentation
  • Enhancing workflow documentation
  • Next steps
• Part 3. Data dictionaries and lineage graphs
  • What we can generate
  • Output Formats and Modes
  • Running the tool
  • Exploring the Artifacts
  • Enriching the Data Dictionary
  • Lineage Diagram for Medicare data

Runnable copies of the finished workflow and data model are provided in examples/with-postgres/climate/; the simpler file-only (no database) variant of the workflow is described in Example: aggregating a climate variable.