kathryn_cowie
07/18/2022, 2:36 PMtaylor
07/18/2022, 3:17 PMchristoph
07/18/2022, 10:35 PMA lot of folks do this formatting in the warehouseJust quickly chiming in here with a "This is how we do it" comment: • Meltano ELT runs from SaaS sources (many different) into Postgres. And DBT models manage the different "shapes" of analytics exposures according to requirements (i.e. some models do the analytics insights transformations into the traditional datawarehouse "shapes". And then other dedicated dbt models have the task of shaping the necessary JSON objects that have the requisite format to be loaded back into external data stores. • Postgres happens to have fantastic support for working with JSON (both parsing and generating it) and makes this task very easy. • At the end, meltano does the reverse EL step of reading the Postgres relations with the correctly shaped data in it and dump that into a JSONL file for actual loading into the external system.
kathryn_cowie
07/27/2022, 2:52 AMchristoph
07/27/2022, 7:16 AMkathryn_cowie
07/27/2022, 3:24 PMtarget-hubspot
and in the mean time work on getting closely acquainted with DBT