has anyone switched from a database to just parque...
# announcementsl
lance_dacey
09/10/2019, 12:13 PMhas anyone switched from a database to just parquet files on S3 or blob? maybe Spark or Apache Drill to create views/final clean tables? sad thing is that dbt is not supported which I really like a lot
m
micael_bergeron
09/10/2019, 12:33 PMthat is an interesting problem, I know that some taps support the VERSION message, i.e. create 1 table schema per VERSION. Then you'd have to create transforms that merge all the schemas into one, canonical schema. That's a lot of work.
l
lance_dacey
09/10/2019, 12:35 PMyeah - ultimately I want all raw data and the only schema that I want to be personally aware of is the Domo one (which is a simple json colname, coltype). so basically, a read schema where I select specific columns
lance_dacey
09/10/2019, 12:36 PMwith the database now, I simply won't have new columns on the table unless I alter the table and rerun the ETL from whatever date the column was added
lance_dacey
09/10/2019, 12:36 PMthat's mainly because we use a database migration tool so every change (even adding a column) is done a bit manually
m
micael_bergeron
09/10/2019, 12:36 PMAs for Parquet (and most columnar file formats) we could write a target for it, then it would be you responsibility to merge the files afterwards.
4 Views