https://linen.dev logo
#plugins-general
Title
# plugins-general
f

fred_reimer

10/22/2021, 1:33 PM
I did a bit of searching, but didn't find anything relevant quickly. Can we configure multiple DBT transforms that are kicked off in sequence, or in parallel if desired, in a single Meltano ELT run? It looks like we can configure multiple transforms in the same meltano.yml file. Sounds like the transform run will be the same name as the extrator (tap-whatever) pulled from package dbt-tap-whatever. Conceptually the goal is to separate the transform of an input from a tap into canonical form from the transform of canonical data for analysis. For instance, we may, and do, have multiple inputs for a category of data, but the format from the various endpoints is all different. In keeping with the ELT philosophy, we would extract and load as close to the original source format as possible, then transform. We would define a canonical form for the data, and the initial transform would be from the disparate original forms into the canonical. Then, we would also need to do analytics on the canonical data. The analysis performed may differ over time based on business needs, but the transformation of the original data to canonical form will likely change much less often, if at all. That is one of the many reasons for keeping these transforms separate. How are others handling this?