https://linen.dev logo
#singer-tap-development
Title
# singer-tap-development
s

stephen_bailey

10/26/2021, 2:14 AM
For that tap, I also implemented some custom data tests that might be worth refining/abstracting for the SDK. The approach was this: 1. In a Pytest fixture, perform a full tap sync with the sample config. 2. Read stdout and parse the records into an array. Then group the records by TYPE and STREAM. 3. Create a generic set of tests that can be applied on a stream basis: at least one record returned, catalog schema keys are in the record schema and vice versa. 4. Apply the generic tests for each stream, passing in the parsed full sync results. This approach allowed me to catch several schema mismatches and a few critical issues related to the state partitioning keys mentioned above. And if it was part of the SDK, it would require almost no code to run on every stream. See: https://github.com/MeltanoLabs/tap-slack/blob/7892c39667f7817e426ee025d2c52622568c38d6/tests/test_streams.py#L27