Universal uploader of the tabular data to the database

Source code

Tool. Runs: dorieh.platform.loader.data_loader

Description

This tool ingests tabular data, usually in CSV format into the database

Inputs

Name	Type	Default	Description
registry	File		A path to the data model file
table	string		the name of the table to be created
database	File		Path to database connection file, usually database.ini
connection_name	string		The name of the section in the database.ini file
domain	string
input	[‘File’, ‘File[]’]		A path the downloaded data files
pattern	string	*.csv*
threads	int	4	number of threads, concurrently writing into the database
page_size	int	1000	explicit page size for the database
log_frequency	long	100000	informational logging occurs every specified number of records
limit	long?		if specified, the process will stop after ingesting the specified number of records
depends_on	Any?		a special field used to enforce dependencies and execution order

Outputs

Name	Type	Description
log	File
errors	stderr