clean-iron-9432212/15/2020, 10:23 PM
ripe-musician-5993312/16/2020, 3:49 PM
What tap do you recommend I look at as an example if I want to implement one that supports dicovery, logging and all the other nice extras?As we speak, @salmon-actor-23953 is working on a new framework that'll make it really easy to build high-quality fully-featured taps: https://gitlab.com/meltano/meltano/-/issues/2401 / https://gitlab.com/meltano/tap-base/-/merge_requests/1, but that won't be ready for a little while. Until then, I'd suggest starting with https://github.com/singer-io/singer-tap-template/ and taking inspiration from https://github.com/singer-io/tap-stripe and https://github.com/singer-io/tap-shopify, both of which are relatively complex, but cover all of the necessary bits.
Is there any convention/best practice to extend metadata? (I want to add some labels for PII, lineage and other stuff).There isn't currently a best practice or standard, but some targets will add additional metadata columns to their loaded tables prefixed with
(after "Stitch Dot Com"): https://transferwise.github.io/pipelinewise/user_guide/metadata_columns.html You can choose to add metadata columns to the
messages your tap outputs, or to expose a separate metadata stream with metadata relating to records in other streams.