i am testing out some ways of doing standardized d...
# singer-tap-development
s
i am testing out some ways of doing standardized data testing in
tap-slack
and it's working pretty well (e.g., https://github.com/MeltanoLabs/tap-slack/blob/6f6928c01253dd7bad091f5b653528d985fd1060/tests/test_core.py) . But it now has me thinking -- what if we specified tests / data expectations in the stream schema file, and then generated the pytest tests based on the schemas, in a similar way that dbt does? I haven't evaluated it deeply but it looks like pytest supports this (https://docs.pytest.org/en/latest/how-to/parametrize.html#pytest-generate-tests). just seems like it would be very neat to write
th.Property("updated_at", th.DateTimeType, expectations=["is_valid_timestamp", "not_null"])
and pytest takes care of the rest. just thinking out loud here
It looks like this could be done by, instead of running
test_channels_stream
, it woudl be
test_stream_primary_keys
and searching through the tap to find all streams that have that expectation. Similarly, it would be
test_stream_attribute_uniqueness
and searching for all the attributes across all streams that need the test. Then that one pytest would be parameterized to go through all of those attributes.