This is still a WIP but for anyone interested I wrote a Singer tap for dbt artifacts. While it should work with other targets, I wrote it primarily with the intention of loading this data to to a BigQuery target. Because I wanted to load each field into BigQuery’s native types instead of just dumping the JSON to a string column, I had to throw in some transformation logic. For example, some artifacts have fields that are of type object with a property for each column in your table - I transformed that to instead be an array with one entry per column so that my final result has a definite set of columns in it. https://github.com/prratek/tap-dbt-artifacts

Interesting, kinda your own version of

dbt docs

I wonder about doing this for the catalog files generated from running

meltano invoke

/

meltano elt

interesting! this should be pretty useful. I see only one record is created from the run_results file. What do you think about making one record per model perhaps as parent-child streams?

@visch i’m working on building a data dictionary/search app for our Looker instance i thought being able to enrich that with test results and freshness info could be compelling, so that’s the immediate use case

no you're right, leaving the unnesting to the T layer leads to fewer bugs and incremental replication for child streams can be tricky. How about unnesting (part of) the metadata to make the

invocation_id

a top level field so it can be used as a primary key?

that makes sense. the table is missing a top level primary key as is