Dorieh Data Platform

Overview

  • Introduction
    • Introduction to Data Platform
    • Building Blocks
      • Dorieh Utilities
      • Core Platform
      • Dorieh GIS Utilities
      • Dorieh Documentation Utilities
      • Data Processing and Loading Pipelines
    • Working with NSAPH containerized apps
    • Deployment
    • Using the Database
    • Terms and Acronyms
    • Building Platform documentation
    • Next steps
  • Why a data platform
    • Why we need a Data Platform
    • Architecture
    • Supported Programming Languages and Tools
    • Development Mode
    • Where it can be deployed
    • What makes Dorieh different
  • The Dorieh approach
    • Two languages, one pipeline
    • Medallion architecture as Dorieh implements it
    • Dataset operators and field construction operators
    • Families of field transformations
    • Disambiguation rules
    • Validation and journaling
    • Fine-grained lineage
    • Where each concept lives
  • The companion book
    • Bibliographic information
    • Chapter-to-documentation map
    • Using this documentation without the book

Learning paths

  • Choose your path
    • Researchers and analysts
      • The path
      • Going deeper
    • Pipeline developers
      • The path
      • Going deeper
    • Platform operators
      • The path
      • Going deeper
    • Readers of the book
      • The path
      • Going deeper
    • Reviewers and auditors
      • The path
      • Going deeper

Data domains and pipelines

  • Data Domains
    • Health
      • Health Data in Dorieh (Medicare and Medicaid)
      • Medicare: Building a Data Warehouse from ResDac Files
      • Medicaid: Building a Data Warehouse from ResDac Files
      • See also
    • Climate
      • What is gridMET?
      • Using command line gridMET utility
      • Example
      • Python modules
      • CWL pipelines and tools
    • Exposure (PM2.5, WashU)
      • Python packages
      • CWL Workflows
    • Environmental Protection Agency (EPA) data
      • Package Contents
      • Querying Data
    • Demographics
      • About
      • Setting Up Environment
      • Quick Start
      • Census Variable File Structure
      • Stored Data Files
      • Modules
      • Glossary
      • Working with the census Package
      • census_assemble.cwl
      • census_assemble.cwl
      • census_density.cwl
      • census_density.cwl
      • census_ingest.cwl
      • Sub-workflow store from census_ingest.cwl
      • CWL sub-workflow for step store of workflow census_ingest.cwl
      • census_ingest.cwl
      • census_interpolate.cwl
      • census_interpolate.cwl
      • census_qc.cwl
      • census_qc.cwl
      • census_workflow.cwl
      • census_workflow.cwl
      • census_write.cwl
      • census_write.cwl
      • Indices and tables
  • Data Processing Pipelines
    • Introduction
    • Workflow Description Languages
    • Running Workflows
      • Tested Runners
      • Providing Parameters to the Pipelines
      • Using Toil
      • Troubleshooting Workflows Run by Toil
    • Testing Workflows
    • Published and Tested Workflows
      • Pipeline to aggregate data from Climatology Lab
      • Pipeline to ingest Monthly Pollution data downloaded from WashU Box
      • Pipeline to aggregate data in NetCDF format over given geographies
      • Full EPA AQS Processing Pipeline
      • Full EPA AirNow Processing Pipeline (including downloading shapefiles)
      • Medicare data ingestion and processing pipeline
      • Full Medicaid Processing Pipeline
      • census_workflow.cwl
    • Developing New Workflows
      • Combining Included CWL Tools into a New Workflow
      • Wrapping Python Modules as CWL Tools

Tutorials and worked examples

  • Dorieh Tutorials
    • Climate tutorial
      • Part 1. Building the pipeline
      • Part 2. Documenting the workflow
      • Part 3. Data dictionaries and lineage graphs
    • Medicare claims pipeline tutorial
      • Introduction
      • Designing the pipeline
      • From FTS documents to machine-readable schemas
      • Bronze: one table per source file
      • Silver: harmonize, disambiguate, validate
      • Gold: the QC aggregates
      • Orchestration: five steps in medicare.cwl
      • Documentation and lineage
      • Running it yourself
  • Examples
    • A CWL workflow example: aggregating a climate variable
      • What the sample workflow is doing: aggregating a climate variable
      • Prepare to run a workflow
      • Running the workflow in Python virtual environment
      • Running the workflow using Docker
    • Medicare pipeline with publicly available synthetic data (PostgreSQL + Superset)
      • Prerequisites
      • Step 1 — Download synthetic sample data
      • Step 2 — Run the Medicare processing pipeline
      • Step 3 (Optional) — Explore results in Apache Superset
      • What the pipeline produces
    • Related how-to guides

Data dictionaries

  • Data Dictionaries
    • Medicare data warehouse
      • Table Lineage Diagram
      • Alphabetic list of all tables
      • Alphabetic list of all columns in all tables
      • The Data Dictionary Generation tool

Data modeling DSL reference

  • Data Modeling for Dorieh Data Platform
    • Data Modeling Extensions: Federated Views Across Years
      • Combining multiple sources and optional columns
      • Exclude
      • Cast
      • Validating consistency of data across tables
    • Introduction to data modeling for Dorieh Data Platform
    • Domain
    • Table
      • Create statement
      • Invalid Record
    • Column
      • Source
      • Index
      • Generated columns
      • Computed columns
      • File columns
      • Record columns
      • Identifier columns and the {identifiers} token
      • Transposing columns
      • Wildcards
    • Multi-column indices
    • Indexing policies
    • Linking with nomenclature
      • US States
      • County codes
    • Ingesting data
    • Where to go next

Platform

  • Python Packages
    • General purpose utilities
      • Overview of Utilities
      • Current Development
      • TODO
      • Documentation Indices
    • Data platform components
      • Package dorieh.platform
      • Package dorieh.platform.data_model
      • Package dorieh.platform.loader
      • Package dorieh.platform.requests
      • Package dorieh.platform.utils
    • GIS utilities
      • GIS Library Overview
      • Shape files Downloader
    • Health data manipulation tools
      • Package cms
      • Package dorieh.cms.tools
      • Additional details
    • Raster data tools (climate and exposure)
      • General tools for raster files
      • Specialized tools for working with climate data
      • Specialized tools for working with exposure data
  • Data Platform Internals
    • Dorieh Core Data Platform
      • Managing database connections
      • Dorieh Data Loader
      • Project (Directory) Loading Utility
      • Mapping between different territorial codes
      • SQL Documentation
      • Core platform overview
      • Tool Examples
      • Project Structure
      • Documentation Indices
    • Dorieh Deployment
      • Using Docker
      • Using Python Virtual Environment
      • Using Conda Package Manager
      • Using Toil with Docker requirement
  • Platform capabilities
    • How to query the database
      • Setup
      • Create connection definition file
      • Executing the query
      • Using EXPLAIN to optimize queries
    • Querying Medicaid Data
      • Querying diagnoses
    • Monitoring database activity
      • Module monitor
      • CLI Usage for monitor
      • Using as API
    • Approximate distinct counting with HLL
    • Handling user requests
      • Introduction
      • Structure
    • Example user request

Operational guides

  • Database Testing Framework
    • Utility to generate test queries
      • Test runner: Utility to run generated tests
      • The gen_dbt_cwl Module
      • Usage
      • Details
    • Test runner: Utility to run generated tests
      • Usage
      • Details
    • The gen_dbt_cwl Module
      • CWLTestGenerator
    • Testing bundled workflows
      • Introduction to testing and prerequisites
      • Before running any test
      • Testing AQS workflow
      • Testing Airnow workflow
      • Testing Climate workflow
      • Testing Exposure workflow
  • Adding more data
    • What data are you adding?
    • Data modeling vs data introspection
    • Adding new data domain
    • Creating new single table
    • Adding a table from a remote data source
    • Adding data to existing table
    • Automatically ingesting multiple files from a file system
  • Executing containerized apps
    • Introduction
    • Prerequisites
      • Docker or Python virtual environment
      • Setup DBMS Server
      • Define connection
    • Using pipeline generator
      • Generate pipeline and metadata
    • Execute generated pipeline
    • Appendix 1: Metadata description
      • File app.config.yaml
      • File metadata.yml
      • File dorieh-metadata.yaml

Reference

  • Terms and Acronyms
    • Concepts
    • Acronyms
  • Indices
Dorieh Data Platform
  • Python Packages Included in the Platform
  • Computational Utilities for working with raster data
  • View page source

Computational Utilities for working with raster data

General tools for raster files

  • The display_netcdf Module
  • The prof Module
    • ProfilingData
  • The config Module
    • DateFilter
    • Shape
    • GridmetVariable
    • OutputType
    • GridContext
    • GridMETContext

Specialized tools for working with climate data

Computational Utilities for working with Climate gridMET data

Specialized tools for working with exposure data

Exposure Data from the Atmospheric Composition Analysis Group at Washington University in St. Louis

Previous Next

© Copyright 2021-2024, Harvard University.

Built with Sphinx using a theme provided by Read the Docs.