Anyone using Meltano also using things like Apache Iceberg and Delta Tables? I ask because I am pretty big noob in the structured data space and trying to see how best to implement one of these types of things but not seeing how/if it would be helped or hindered by DBT / Meltano Targets ... like is there a way to just code my target to auto-add-delta tables or.. where does the meltano team see things like this fitting into the picture? Thanks citations https://github.com/delta-io/delta-sharing https://iceberg.apache.org/

Did you find anyone yet? I'm very interested. In general it's not a problem to write a target to create delta tables or Iceberg tables. In fact I like the idea of going through a table format, it abstracts away a piece of the storage. I'm wondering whether someone sees even more here, because table formats do come with some nice additional features that might be leveraged by Meltano. CC @aaronsteers /@taylor

The closest I'm aware of in this direction are the

target-parquet

and

target-athena

iterations - since both are aimed at landing parquet/spark-ish data and registering in a metastore. I don't think the parquet target did this, but athena did, and was therefor also (in theory) something that could be made compatible with redshift spectrum, emr, etc.

Thanks Sven and AJ, I currently use the

target-parquet

to produce local

.parquet

files.. eventually those get pushed up to S3.. I do not use Athena and therefore won't be going down that route with it nor it's AWS Glue service.. but I could imagine if a target knew the AWS Glue was in the mix it would be quite easy to push/update delta-ish tables