Meltano

image.png

i am testing out some ways of doing standardized data testing in `tap-slack` and it's working pretty well (e.g., <https://github.com/MeltanoLabs/tap-slack/blob/6f6928c01253dd7bad091f5b653528d985fd1060/tests/test_core.py>) .

But it now has me thinking -- what if we specified tests / data expectations in the stream schema file, and then _generated_ the pytest tests based on the schemas, in a similar way that dbt does? I haven't evaluated it deeply but it looks like pytest supports this (<https://docs.pytest.org/en/latest/how-to/parametrize.html#pytest-generate-tests>). just seems like it would be very neat to write `th.Property("updated_at", th.DateTimeType, expectations=["is_valid_timestamp", "not_null"])`  and pytest takes care of the rest. just thinking out loud here

It looks like this could be done by, instead of running `test_channels_stream`, it woudl be `test_stream_primary_keys` and searching through the tap to find all streams that have that expectation. Similarly, it would be `test_stream_attribute_uniqueness` and searching for all the attributes across all streams that need the test. Then that one pytest would be parameterized to go through all of those attributes.