The task Module

This module contains classes used to define, schedule and execute long running computations used by gridMET pipelines

count_lines(f)

quote(s: str) → str

class Parallel(value)

An enumeration.

points = 'points'

bands = 'bands'

days = 'days'

class Collector

abstract writerow(data: List)

flush()

class CSVWriter(out_stream)

writerow(row: List)

flush()

class ListCollector

writerow(data: List)

get_result()

class ComputeGridmetTask(year: int, variable: GridmetVariable, infile: str, outfile: str, date_filter=None, ram: int = 0)

An abstract class for a computational task that processes data in Unidata netCDF (Version 4) format

Parameters:

• ram –

• date_filter –

• year – year

• variable – Gridemt band (variable)

• infile – File with source data in NCDF4 format

• outfile – Resulting CSV file

origin = datetime.date(1900, 1, 1)

classmethod get_variable(dataset: Dataset, variable: GridmetVariable)

abstract get_key()

get_days() → Dict

prepare() → Dict

on_prepare()

execute(mode: str = 'wt')

Executes computational task

Parameters:

mode (str) – mode to use opening result file

Returns:

collect_data(days: Dict, collector: Collector)

abstract compute_one_day(writer: Collector, day, layer)

Computes required statistics for a single day. This method is called by execute() and is implemented in specific subclasses

Parameters:

• writer – CSV Writer to output the result

• day – day

• layer – layer, corresponding to the day

Returns:

Nothing

to_date(day) → date

class ComputeShapesTask(year: int, variable: GridmetVariable, infile: str, outfile: str, strategy: RasterizationStrategy, shapefile: str, geography: Geography, date_filter=None, ram=0)

Class describes a compute task to aggregate data over geography shapes

The data is expected in .. _Unidata netCDF (Version 4) format: https://www.unidata.ucar.edu/software/netcdf/

Parameters:

• ram –

• date_filter –

• year – year

• variable – gridMET band (variable)

• infile – File with source data in NCDF4 format

• outfile – Resulting CSV file

• strategy – Rasterization strategy to use

• shapefile – Shapefile for used collection of geographies

• geography – Type of geography, e.g. zip code or county

on_prepare()

get_key()

compute_one_day(writer: Collector, day, layer)

Computes required statistics for a single day. This method is called by execute() and is implemented in specific subclasses

Parameters:

• writer – CSV Writer to output the result

• day – day

• layer – layer, corresponding to the day

Returns:

Nothing

class ComputePointsTask(year: int, variable: GridmetVariable, infile: str, outfile: str, points_file: str, coordinates: List, metadata: List, date_filter=None, ram=0)

Class describes a compute task to assign data to a collection of points

The data is expected in .. _Unidata netCDF (Version 4) format: https://www.unidata.ucar.edu/software/netcdf/

Parameters:

• ram –

• year – year

• variable – Gridemt band (variable)

• infile – File with source data in NCDF4 format

• outfile – Resulting CSV file

• points_file – path to a file containing coordinates of points in csv format.

• coordinates – A two element list of column names in csv corresponding to coordinates

• metadata – A list of column names in csv that should be interpreted as metadata (e.g. ZIP, site_id, etc.)

force_standard_api = False

get_key()

on_prepare()

make_point(row) → PointInRaster

execute(mode: str = 'w') → None

Executes computational task

Parameters:

mode (str) – mode to use opening result file

Returns:

compute_one_day(writer: Collector, day, layer)

Computes required statistics for a single day. This method is called by execute() and is implemented in specific subclasses

Parameters:

• writer – CSV Writer to output the result

• day – day

• layer – layer, corresponding to the day

Returns:

Nothing

class DownloadGridmetTask(year: int, variable: GridmetVariable, destination: str)

Task to download source file in NCDF4 format

Parameters:

• year – year

• variable – Gridmet band (variable)

• destination – Destination directory for all downloads

BLOCK_SIZE = 65536

classmethod get_url(year: int, variable: GridmetVariable) → str

Constructs URL given a year and band

Parameters:

• year – year

• variable – Gridmet band (variable)

Returns:

URL for download

target()

Returns:

File path for downloaded data

execute()

Executes the task :return: None

class GridmetTask(context: GridMETContext, year: int, variable: GridmetVariable)

Defines a task to download and process data for a single year and variable Instances of this class can be used to parallelize processing

Parameters:

• context – Configuration object for the pipeline

• year – year

• variable – gridMET band (variable)

classmethod destination_file_name(context: GridMETContext, year: int, variable: GridmetVariable)

Constructs a file name for a given set of parameters

Parameters:

• context – Configuration object for the pipeline

• year – year

• variable – Gridmet band (variable)

Returns:

variable_geography_year.csv[.gz]

classmethod find_shape_file(context: GridMETContext, year: int, shape: Shape)

Finds shapefile for a given type of geographies for the closest available year

Parameters:

• context – Configuration object for the pipeline

• year – year

• shape – Shape type

Returns:

a shape file for a given year if it exists or for the latest year before the given

execute()

Executes the task. First the download subtask is executed unless the corresponding file has already been downloaded. Then the compute tasks are executed

Returns:

None