I saw that there is an issue to add Descriptions to the Prop Meltano #plugins-general

I saw that there is an issue to add Descriptions t...

stephen_bailey

08/14/2021, 2:46 PM

I saw that there is an issue to add Descriptions to the Property type helper. (https://gitlab.com/meltano/sdk/-/issues/159) Does anyone know if there are plans/dreams/challenges with having targets consume this metadata from a tap catalog and load it into the database, in the way that dbt does when creating models? As a tap developer, I would love to better document the data model, but it has traditionally seemed like a bridge to nowhere, so to speak.

aaronsteers

08/14/2021, 4:14 PM

Absolutely!!! Yes, I think we had an office hours discussion with that end to end vision, and that inspired the issue above. Assuming JSON catalogs are populated with the text descriptions, this would enable the target to inject those descriptions into target table/column metadata, but even if the target doesn't do so, I imagine that the catalog could still be leveraged by an orchestrator like Meltano to autogenerated dbt source.yml docs, for instance.

aaronsteers

08/14/2021, 4:19 PM

The work on the Tap SDK side is fairly trivial - since we only need to light up a pattern to populate description into the JSON schema. It would then be up to the target developers to use that metadata during target table creation to populate column metadata if the destination supports it. Or, barring broad support in targets, it might be worthwhile to automate the dbt first layer, since that would only need to be written once, and is the driver to

dbt docs

content. Dbt docs is a common way to share project docs between developers and users alike so I think that alone would be a big win.

stephen_bailey

08/14/2021, 7:18 PM

Awesome! Love that this is on the roadmap. Would be a great opportunity to also implement a

tags

property, which could also be used by Meltano for selection:

meltano select tap-salesforce '!tag:pii'

, in development environments,for example. I think the incentive for a target implementing

tag

functionality may be even higher, given the emergence of features like "object tags" in snowflake and "policy tags" in Bigquery.

stephen_bailey

08/14/2021, 7:21 PM

But I do also agree that a

meltano select tap-foo --list --output-format dbt

type of workflow would be a really great first step, although there may be challenges around mapping variant-type data into the actual schema created in the target (?)

Open in Slack

Previous Next