Data Tests for Taps (Didn't want to blow up https://meltano.slack.com/archives/C01PKLU5D1R/p1635450531125300 )

i am doing this now! literally testing out dbt like tests last night

nice, are you using a container to spin up a local postgres?

no, running a limited sync and doing data tests on the records coming in: https://gitlab.com/meltano/sdk/-/merge_requests/197

Does

meltano elt tap-name target-postgres --transform dbt:test

work? I could just run

meltano dbt:test

as well

are you trying to run tests as part of the tap development, or post-sync?

tap development actually. But I was thinking the overlap here with dbt is large, if the overhead isn't too much why not just do the same test I'd do in production?

because you'd have to test after load somewhere, and the load configuration could change the way the data is represented

True, you're coupling your testing to something you don't control.

target-*

, and

dbt

but i think the ergonomics of the dbt testing are right -- the tests should just live with the schema definition. it's very natural to think of this;

th.Property("updated_at", th.DateTimeType, expectations=["is_valid_timestamp", "not_null"])

and then run

pytest

and it runs the

not_null

and

is_valid_timestamp

tests

Schema tests, that's nice! My use case at the moment is data tests. ie

"select (*) from tasks where archived = 'true'

count should be >= 1

is that a custom data test in dbt?

I haven't wrote it yet, was thinking about just doing that as a dbt test yes!

with the MR above, you could write:

def test_archived_tasks_greater_than_zero(test_util):
    records = self.records["tasks"]
    assert len(r for r in records if r["archived"]) > 0

The work @florian.hines has been doing will enable this workflow soon! (I know I keep promising - it’s coming!)

Nice! @stephen_bailey that looks pretty nice @taylor what do you think about runing tests right in the tap for tap developers

with dbt? that’s harder to imagine. I could maybe see great expectatinos… are you thinking this is part of the tap codebase?

I think it's a bit nutty, after @stephen_bailey called me on how much easier it'd be not to But yeah, in the tests dir, throw a meltano elt tap-name target-postgres, then run your dbt tests against it

I’d almost rather have a SQLite or DuckDB instance

dbt no likey

😬

https://meltano.slack.com/archives/C01PKLU5D1R/p1635520594130900?thread_ts=1635519888.127100&cid=C01PKLU5D1R is a pretty good argument not to

so i was playing around with this weekend, and there's a really nice pattern that could be implemented to build "self-testing taps". basically, all tests could be based off of the tap / stream / schema attributes, and dynamically generated at test run time, with each test being run as a separate test. it's pretty neat! currently have it implemented as

test_manifest

array and a

pytest.mark.parameterize

decorator. the test_manifest is a list of

test_name

and `params`:

TEST_MANIFEST = [
    ("tap__cli", {}),
    ("tap__discovery", {}),
    ("tap__stream_connections", {}),
    ("stream__catalog_schema_matches_record", {"stream_name": "channels"}),
    ("stream__record_schema_matches_catalog", {"stream_name": "channels"}),
    ...
    ("stream__returns_record", {"stream_name": "users"}),
    ("stream__primary_key", {"stream_name": "users"}),
    ("attribute__unique", {"stream_name": "channels", "attribute_name": "id"}),
    ("attribute__not_null", {"stream_name": "channels", "attribute_name": "id"}),
]

and then you parameterize a single function that pulls the test function from the test utility and runs it.

@pytest.mark.parametrize(
    "test_config", TEST_MANIFEST
)
def test_builtin_tap_tests(test_util, test_config):
    test_name, params = test_config
    test_func = test_util.available_tests[test_name]
    test_func(**params)

will carve out some time and demo it next week. it works surprisingly well

Curious stuff! Still debating how I want to test my tap 😕

what are the main things you want to test?