Full EPA AirNow Processing Pipeline (including downloading shapefiles)
Workflow
Description
This workflow downloads AirNow data from the government servers, introspects it to infer the database schema and ingests the data into the database
Starting with version 0.5.0, the workflow initializes the database (via initcoredb.cwl) before ingesting the data, hence it can be run against a pristine, empty database. Earlier versions required the database to be initialized beforehand (e.g., by running another pipeline first).
Example run:
cwl-runner airnow.cwl sample_airnow.yml
Or
cwl-runner --parallel /opt/airflow/project/epa/src/cwl/airnow.cwl --database /opt/airflow/project/database.ini --connection_name nsaph2 --proxy $HTTP_PROXY --api-key XXXXXXXX-YYYY-ZZZZ-XXXX-YYYYYYYYY --from 2022-01-01 --to 2022-08-31 --parameter_code pm25 --table airnow_pm25_2022
Inputs
Name |
Type |
Default |
Description |
|---|---|---|---|
proxy |
string? |
HTTP/HTTPS Proxy if required |
|
api-key |
string |
API key for AirNow |
|
database |
File |
Path to database connection file, usually database.ini |
|
connection_name |
string |
The name of the section in the database.ini file |
|
from |
string |
Start date for downolading, in YYYY-MM-DD format |
|
to |
string |
End date for downolading, in YYYY-MM-DD format |
|
parameter_code |
string |
Parameter code. Either a numeric code (e.g. 88101, 44201) or symbolic name (e.g. PM25, NO2). See more: AQS Code List |
|
table |
string |
Name of the table to be created in the database |
|
year |
int |
Outputs
Name |
Type |
Description |
|---|---|---|
shapes_data |
File[] |
|
download_log |
File |
|
ingest_log |
File |
|
index_log |
File |
|
vacuum_log |
File |
|
download_data |
File |
|
model |
File |
Steps
Name |
Runs |
Description |
|---|---|---|
get_shapes |
This step downloads Shape files from a given collection (TIGER/Line or GENZ) and a geography (ZCTA or Counties) from the US Census website, for a given year or for the closest one. |
|
download |
||
introspect |
||
initdb |
Ensure that database utilities are at their latest version |
|
ingest |
Uploads data into the database |
|
index |
||
vacuum |