Hello everyone! Our team is creating a custom tap and we will be providing descriptions for all entities and attributes. Our pipeline will be using a snowflake-target and we would like for these descriptions to populate as comments for the Snowflake tables the target creates/updates. It doesn't look like any of the snowflake targets support this flow, so we'll be creating a fork with this functionality. Before we do so, I just wanted to check with the community to confirm that there wasn't another well established way of accomplishing this transfer of description metadata from tap to target, and furthermore, that our planned approach doesn't conflict with other aspects of the platform.

We discussed this in an office hour a while back, I gave it a stab on target-oracle but it seems I never merged it to main. You might get some inspiration here: https://github.com/radbrt/target-oracle/tree/col_comments Perhaps in particular, the CI test based on this file: https://github.com/radbrt/target-oracle/blob/col_comments/target_oracle/tests/data_files/column_comments.singer

Excellent, this is exactly what we had in mind. Thanks for the confirmation!

Hey @oren_ely, this does not exactly answer your question, but if you are a dbt user, you can use the persist docs flag to automatically document downstream tables. Hope this helps 😄

Thanks! We actually were looking at this as well for our transformations. The documentation characterizes this as a way to persist descriptions from the dbt yml files, but if it also propagates comments from source tables to downstream model tables that would be very helpful.

@oren_ely from what I understand it does not do that though - I would recommend doc blocks or dbt osmosis for that

dbt-osmosis looks promising! Thanks, we'll look into it.