Dorieh Data Platform

Overview

  • Introduction
    • Introduction to Data Platform
    • Building Blocks
      • Dorieh Utilities
      • Core Platform
      • Dorieh GIS Utilities
      • Dorieh Documentation Utilities
      • Data Processing and Loading Pipelines
    • Working with NSAPH containerized apps
    • Deployment
    • Using the Database
    • Terms and Acronyms
    • Building Platform documentation
    • Next steps
  • Why a data platform
    • Why we need a Data Platform
    • Architecture
    • Supported Programming Languages and Tools
    • Development Mode
    • Where it can be deployed
    • What makes Dorieh different
  • The Dorieh approach
    • Two languages, one pipeline
    • Medallion architecture as Dorieh implements it
    • Dataset operators and field construction operators
    • Families of field transformations
    • Disambiguation rules
    • Validation and journaling
    • Fine-grained lineage
    • Where each concept lives
  • The companion book
    • Bibliographic information
    • Chapter-to-documentation map
    • Using this documentation without the book

Learning paths

  • Choose your path
    • Researchers and analysts
      • The path
      • Going deeper
    • Pipeline developers
      • The path
      • Going deeper
    • Platform operators
      • The path
      • Going deeper
    • Readers of the book
      • The path
      • Going deeper
    • Reviewers and auditors
      • The path
      • Going deeper

Data domains and pipelines

  • Data Domains
    • Health
      • Health Data in Dorieh (Medicare and Medicaid)
      • Medicare: Building a Data Warehouse from ResDac Files
      • Medicaid: Building a Data Warehouse from ResDac Files
      • See also
    • Climate
      • What is gridMET?
      • Using command line gridMET utility
      • Example
      • Python modules
      • CWL pipelines and tools
    • Exposure (PM2.5, WashU)
      • Python packages
      • CWL Workflows
    • Environmental Protection Agency (EPA) data
      • Package Contents
      • Querying Data
    • Demographics
      • About
      • Setting Up Environment
      • Quick Start
      • Census Variable File Structure
      • Stored Data Files
      • Modules
      • Glossary
      • Working with the census Package
      • census_assemble.cwl
      • census_assemble.cwl
      • census_density.cwl
      • census_density.cwl
      • census_ingest.cwl
      • Sub-workflow store from census_ingest.cwl
      • CWL sub-workflow for step store of workflow census_ingest.cwl
      • census_ingest.cwl
      • census_interpolate.cwl
      • census_interpolate.cwl
      • census_qc.cwl
      • census_qc.cwl
      • census_workflow.cwl
      • census_workflow.cwl
      • census_write.cwl
      • census_write.cwl
      • Indices and tables
  • Data Processing Pipelines
    • Introduction
    • Workflow Description Languages
    • Running Workflows
      • Tested Runners
      • Providing Parameters to the Pipelines
      • Using Toil
      • Troubleshooting Workflows Run by Toil
    • Testing Workflows
    • Published and Tested Workflows
      • Pipeline to aggregate data from Climatology Lab
      • Pipeline to ingest Monthly Pollution data downloaded from WashU Box
      • Pipeline to aggregate data in NetCDF format over given geographies
      • Full EPA AQS Processing Pipeline
      • Full EPA AirNow Processing Pipeline (including downloading shapefiles)
      • Medicare data ingestion and processing pipeline
      • Full Medicaid Processing Pipeline
      • census_workflow.cwl
    • Developing New Workflows
      • Combining Included CWL Tools into a New Workflow
      • Wrapping Python Modules as CWL Tools

Tutorials and worked examples

  • Dorieh Tutorials
    • Climate tutorial
      • Part 1. Building the pipeline
      • Part 2. Documenting the workflow
      • Part 3. Data dictionaries and lineage graphs
    • Medicare claims pipeline tutorial
      • Introduction
      • Designing the pipeline
      • From FTS documents to machine-readable schemas
      • Bronze: one table per source file
      • Silver: harmonize, disambiguate, validate
      • Gold: the QC aggregates
      • Orchestration: five steps in medicare.cwl
      • Documentation and lineage
      • Running it yourself
  • Examples
    • A CWL workflow example: aggregating a climate variable
      • What the sample workflow is doing: aggregating a climate variable
      • Prepare to run a workflow
      • Running the workflow in Python virtual environment
      • Running the workflow using Docker
    • Medicare pipeline with publicly available synthetic data (PostgreSQL + Superset)
      • Prerequisites
      • Step 1 — Download synthetic sample data
      • Step 2 — Run the Medicare processing pipeline
      • Step 3 (Optional) — Explore results in Apache Superset
      • What the pipeline produces
    • Related how-to guides

Data dictionaries

  • Data Dictionaries
    • Medicare data warehouse
      • Table Lineage Diagram
      • Alphabetic list of all tables
      • Alphabetic list of all columns in all tables
      • The Data Dictionary Generation tool

Data modeling DSL reference

  • Data Modeling for Dorieh Data Platform
    • Data Modeling Extensions: Federated Views Across Years
      • Combining multiple sources and optional columns
      • Exclude
      • Cast
      • Validating consistency of data across tables
    • Introduction to data modeling for Dorieh Data Platform
    • Domain
    • Table
      • Create statement
      • Invalid Record
    • Column
      • Source
      • Index
      • Generated columns
      • Computed columns
      • File columns
      • Record columns
      • Identifier columns and the {identifiers} token
      • Transposing columns
      • Wildcards
    • Multi-column indices
    • Indexing policies
    • Linking with nomenclature
      • US States
      • County codes
    • Ingesting data
    • Where to go next

Platform

  • Python Packages
    • General purpose utilities
      • Overview of Utilities
      • Current Development
      • TODO
      • Documentation Indices
    • Data platform components
      • Package dorieh.platform
      • Package dorieh.platform.data_model
      • Package dorieh.platform.loader
      • Package dorieh.platform.requests
      • Package dorieh.platform.utils
    • GIS utilities
      • GIS Library Overview
      • Shape files Downloader
    • Health data manipulation tools
      • Package cms
      • Package dorieh.cms.tools
      • Additional details
    • Raster data tools (climate and exposure)
      • General tools for raster files
      • Specialized tools for working with climate data
      • Specialized tools for working with exposure data
  • Data Platform Internals
    • Dorieh Core Data Platform
      • Managing database connections
      • Dorieh Data Loader
      • Project (Directory) Loading Utility
      • Mapping between different territorial codes
      • SQL Documentation
      • Core platform overview
      • Tool Examples
      • Project Structure
      • Documentation Indices
    • Dorieh Deployment
      • Using Docker
      • Using Python Virtual Environment
      • Using Conda Package Manager
      • Using Toil with Docker requirement
  • Platform capabilities
    • How to query the database
      • Setup
      • Create connection definition file
      • Executing the query
      • Using EXPLAIN to optimize queries
    • Querying Medicaid Data
      • Querying diagnoses
    • Monitoring database activity
      • Module monitor
      • CLI Usage for monitor
      • Using as API
    • Approximate distinct counting with HLL
    • Handling user requests
      • Introduction
      • Structure
    • Example user request

Operational guides

  • Database Testing Framework
    • Utility to generate test queries
      • Test runner: Utility to run generated tests
      • The gen_dbt_cwl Module
      • Usage
      • Details
    • Test runner: Utility to run generated tests
      • Usage
      • Details
    • The gen_dbt_cwl Module
      • CWLTestGenerator
    • Testing bundled workflows
      • Introduction to testing and prerequisites
      • Before running any test
      • Testing AQS workflow
      • Testing Airnow workflow
      • Testing Climate workflow
      • Testing Exposure workflow
  • Adding more data
    • What data are you adding?
    • Data modeling vs data introspection
    • Adding new data domain
    • Creating new single table
    • Adding a table from a remote data source
    • Adding data to existing table
    • Automatically ingesting multiple files from a file system
  • Executing containerized apps
    • Introduction
    • Prerequisites
      • Docker or Python virtual environment
      • Setup DBMS Server
      • Define connection
    • Using pipeline generator
      • Generate pipeline and metadata
    • Execute generated pipeline
    • Appendix 1: Metadata description
      • File app.config.yaml
      • File metadata.yml
      • File dorieh-metadata.yaml

Reference

  • Terms and Acronyms
    • Concepts
    • Acronyms
  • Indices
Dorieh Data Platform
  • Data Dictionaries
  • Data dictionary and lineage for Medicare processing
  • Table Lineage Diagram
  • View medicare._beneficiaries
  • Column medicare._beneficiaries.orec
  • Column medicare._beneficiaries.orec Lineage SVG
  • View page source

Column medicare._beneficiaries.orec Lineage SVG

medicare._beneficiaries.orec medicare._beneficiaries.orec Original Reason for Entitlement Code (OREC). Set at the time of enrollment and invariant for the life of the beneficiary, hence a beneficiary-level attribute. The canonical value is taken from the earliest enrollment year (ties broken by smallest code, for reproducibility). If the raw data is inconsistent across years the divergence is surfaced via orec_latest / consistent_orec rather than silently affecting joins. (array_agg(orec ORDER BY year, orec))[1] medicare._ps.orec medicare._ps.orec medicare._ps.orec->medicare._beneficiaries.orec  Aggregated On bene_id medicare.ps.orec medicare.ps.orec medicare.ps.orec->medicare._ps.orec Copied medicare.ps.orec_parent 8 incoming links (columns) medicare.ps.orec_parent->medicare.ps.orec
Previous Next

© Copyright 2021-2024, Harvard University.

Built with Sphinx using a theme provided by Read the Docs.