Utility to generate test queries

Generates SQL script that can be used with Test runner: Utility to run generated tests. These scripts are used by The gen_dbt_cwl Module.

Usage

python -m dorieh.platform.dbt.create_test --script SCRIPT [SCRIPT ...]
      [--table TABLE]
      [--db DB]
      [--connection CONNECTION]
      [--autocommit]
      [-h]
      [--verbose]

Options:

Options

Option	Alias	Description
--help	-h	Show this help message and exit
--script SCRIPT [SCRIPT...]	-s SCRIPT [SCRIPT...]	Path to the file to write the test scripts
--autocommit		Use autocommit, default: False
--database DB	--db DB	Path to a database connection parameters file, default: database.ini (in the working directory
--connection CONNECTION	--connection_name CONNECTION	Section in the database connection parameters file
--table TABLE	-t TABLE	Name of the table to being tested, default: None
--verbose		Verbose output, default: False

Details

A utility to generate a test based on a sample table.

The tool introspects a table, given as an input and generates a set of queries each of which tests that the data in a certain column has not changed.

The queries are output into a file that can be executed as a single SQL query producing a table with the following columns:

1. Name of the column being tested

2. What value is being tested not to change: MD5 hash, number of distinct records, mean value or variance

3. Whether the value has changed (indicated by string failed) or remained the same (indicated by string passed)

Individual queries are separated by a comment strings:

• – Test case end

• – Test case start

so a test runner can execute them individually if desired

Note that the generated queries pin the expected values (MD5 hashes, distinct counts, means and variances) to the table content at the time of generation: if an upstream data source legitimately revises its data, previously generated test scripts become stale and must be regenerated by re-running this tool against a freshly loaded database.

class CType(*values)

categorical = 'categorical'

text = 'text'

integral = 'integral'

numeric = 'numeric'

date = 'date'

class Column(name: str, ctype: CType, is_indexed: bool)

class TableFingerprint(context: DBTConfig = None)

CATEGORICAL_THRESHOLD = 24

get_columns()

get_categories() → List[Column]

generate_tests()

test_column(c: Column)

test_case_sql(name: str, test: str, condition: str) → str

test_exact(sql: str, name: str, test: str) → str

test_approximate(sql: str, name: str, test: str) → str

union()

write_test_script()