Dorieh Data Platform

Contents

  • Introduction
  • What is Data Platform
  • Data Domains
  • Data Processing Pipelines
  • Python Packages
  • Data Modelling for Dorieh Data Platform
  • Examples
  • Dorieh Tutorials
    • Building a data preparation Workflow
    • Documenting a Workflow
  • Data Platform Internals
  • Database Testing Framework
  • Adding more data
  • Executing containerized apps
  • Terms and Acronyms
  • Indices
Dorieh Data Platform
  • Dorieh Tutorials
  • View page source

Dorieh Tutorials

Contents

  • Building a data preparation Workflow
    • Introduction:
    • Prerequisites
    • Design overview
    • Directory layout
    • Step 1. Create a minimal CWL workflow skeleton
    • Step 2. Iteratively Defining Steps and Parameters
    • Step 3. Parameterize for a single day (“toy” run)
    • Step 4. Add database integration (PostgreSQL)
    • Step 5. Building Medallion Layers (Bronze, Silver, Gold)
    • Step 6. Testing the Pipeline
    • Next Steps
  • Documenting a Workflow
Previous Next

© Copyright 2021-2024, Harvard University.

Built with Sphinx using a theme provided by Read the Docs.