Full AirNowProcessing Pipeline (with shapefiles on local file system)
Workflow
Description
This workflow downloads AirNow data from the government servers, introspects it to infer the database schema and ingests the data into the database
Starting with version 0.5.0, the workflow initializes the database (via initcoredb.cwl) before ingesting the data, hence it can be run against a pristine, empty database. Earlier versions required the database to be initialized beforehand (e.g., by running another pipeline first).
Inputs
Name |
Type |
Default |
Description |
|---|---|---|---|
proxy |
string? |
HTTP/HTTPS Proxy if required |
|
api-key |
string |
API key for AirNow |
|
database |
File |
Path to database connection file, usually database.ini |
|
connection_name |
string |
The name of the section in the database.ini file |
|
from |
string |
Start date for downolading, in YYYY-MM-DD format |
|
to |
string |
End date for downolading, in YYYY-MM-DD format |
|
parameter_code |
string |
Parameter code. Either a numeric code (e.g. 88101, 44201) or symbolic name (e.g. PM25, NO2). See more: AQS Code List |
|
table |
string |
Name of the table to be created in the database |
|
shapes |
File[] |
Outputs
Name |
Type |
Description |
|---|---|---|
download_log |
File |
|
ingest_log |
File |
|
index_log |
File |
|
download_data |
File |
|
model |
File |
Steps
Name |
Runs |
Description |
|---|---|---|
download |
||
introspect |
||
initdb |
Ensure that database utilities are at their latest version |
|
ingest |
Uploads data into the database |
|
index |