Introduction to the Dorieh Data Platform

This page surveys the platform’s building blocks, how to deploy it, and how to build this documentation. For the ideas behind the platform, read Why a data platform; for the vocabulary that organizes the rest of the documentation, read The Dorieh approach.

Introduction to Data Platform

Note

Using this documentation with (or without) the book. This documentation is a self-contained companion to the book Research Data that Can Be Trusted (Bouzinier et al., Springer, 2026): both describe the same open-source platform, and no page here requires the book. See About the companion book for the chapter-to-page map and a suggested reading order. Read Why a data platform for the motivation, then The Dorieh approach for the vocabulary that organizes the rest of the documentation.

Dorieh Data Platform is intended for development and deployment of ETL/ELT pipelines that includes complex data processing and data cleansing workflows. Complex workflows require a workflow language, and we have chosen Common Workflow Language (CWL).

We have tested deployment with the following CWL implementations:

The data produced by the data processing workflows is eventually stored in either CSV files, a PostgreSQL DBMS or Parquet files. Dorieh also supports storing results in FST and HDF5 files.

Some of the included data processing workflows use “Extract, Load, Transform,” (ELT) paradigm rather than more traditional “Extract, Transform, Load” ETL. It means that these workflows perform calculations, translations, filtering, cleansing, de-duplicating, validating, and data analysis or summarizations inside a DBMS using DBMS internal tools.

The data platform supports tools written in widely used languages such as Python, C/C++ and Java, R and PL/pgSQL.

A discussion of the aims of this data platform and how reproducible research benefits from it is provided in Why a data platform.

Building Blocks

Dorieh Utilities

The dorieh.utils package is intended to hold python code that will be useful across multiple portions of the Dorieh pipelines.

The included utilities are developed to be as independent of specific infrastructure and execution environment as possible.

Included utilities:

Core Platform

The core platform provides the domain-independent functionality of Dorieh: APIs and command-line utilities that do not depend on any particular data domain, but do assume the infrastructure — a PostgreSQL DBMS (version 13 or later) and a CWL runtime environment.

Some mapping (or crosswalk) tables are also included in the Core Platform module. These tables include mappings between different territorial codes, such as USPS ZIP codes, Census ZCTA codes, FIPS codes for US states and counties, SSA codes for US states and counties. See more information in the Mapping between different territorial codes page.

See also: Managing database connections.

Dorieh GIS Utilities

Per USGS, a Geographic Information System (GIS) is a computer system that analyzes and displays geographically referenced information. It uses data that is attached to a unique location.

This dorieh.gis library contains several modules, aimed to work with US Census shape files. They fall into two categories:

  • Utilities to download appropriate shapefiles for a given geography type and year

  • Utilities to aggregate raster data over given shapefiles

Dorieh Documentation Utilities

Documentation utilities to simplify creation of consistent documentation for Dorieh platform

  • cwl2md Generates Markdown documentation for a CWL tool or workflow

  • collector Generates automatic reStructuredText templates for all Python modules

  • copy_section Copies a specified section from one markdown document to another. This way we can collect summaries in one file

Data Processing and Loading Pipelines

See dedicated Pipelines page for additional details.

Fully tested and supported pipelines are listed in the Pipelines page. At this moment, we have published processing pipelines for all Data Domains except Demographics. Health data pipelines are developed against restricted data, but the repository includes synthetic Medicare data, so the Medicare pipeline can be run and tested end to end; see the Medicare tutorial and its reference, the Medicare case study.

To include additional data in a deployed data-platform instance go to Adding more data section.

Pipelines can be tested with DBT Pipeline Testing Framework

Working with NSAPH containerized apps

National Studies on Air Pollution and Health organization (NSAPH) publishes containerized applications to produce certain types of data. These applications are published on the NSAPH Data Production GitHub.

The Pipeline Generator generates a CWL pipeline to execute the app and ingest the data it produces into Dorieh Data warehouse.

The process of data ingestion consists of two steps:

  1. Generation of the pipeline for data ingestion

  2. Execution of the pipeline

Deployment

Dorieh can be installed as a Python package, run from a prebuilt Docker image, or deployed as a full Docker-Compose stack (described in Why a data platform).

The Python package can be installed with a single command:

pip install dorieh

or, if FST support is desired:

pip install dorieh[FST]

To run workflows one also needs a CWL implementation.

We have tested deployment with the following CWL implementations:

We suggest using Toil. To install Toil just run the following command in your Python Virtual Environment:

pip install "toil[cwl,aws]"

A prebuilt Docker image with Dorieh is available from DockerHub. Pull it to your local machine using

docker pull forome/dorieh

command. The image is built for Intel/AMD and ARM CPUs. ARM architecture is used in AWS Graviton2 processors that, according to AWS, deliver up to 40% better price performance. ARM CPUs are also used by latest Mac computers.

If you would like to modify the container please refer to the README in the docker directory.

Using the Database

To get started with querying the database, see a sample query and a discussion of querying Medicaid health data.

Terms and Acronyms

Included Glossary provides some information about acronyms and other terms used throughout this documentation.

Additionally, General Index and Python Module Index provide direct access to the Dorieh components.

Building Platform documentation

The documentation combines general pages in Markdown format with API pages generated from the source tree; the build_documentation script builds the combined site published on GitHub Pages.

To build documentation:

  1. Clone Dorieh project:

     git clone https://github.com/ForomePlatform/dorieh.git

  2. Cd into the project directory:

     cd dorieh

  3. Create virtual environment (e.g., named .dorieh):

     python -m venv .dorieh

  4. Run build_documentation shell script:

     source .dorieh/bin/activate && ./build_documentation.sh

To integrate Markdown with Sphinx processing we use MyST Parser.

See Documentation Utilities package.

Next steps

Continue with Why a data platform for the motivation behind the platform, then The Dorieh approach for the ideas and vocabulary that organize the rest of this documentation.