https://linen.dev logo
Join Slack
Powered by
Should one be adding custom transforms per tap ins...
# plugins-general
a
Should one be adding custom transforms per tap instead of just manually creating transforms in the dbt project created by the transformers ?
a
Hi, Andrew - I’m not sure I understand you question. Do you mean applying the out-of-box transforms versus making your own? In general, I think most users manually create the dbt transforms they need, since that gives the most control.
a
Right, that’s what I’ve been doing. Just wondering if there’s any reason I should also be running something like 
meltano add --custom transform mytransformname
(there was a link to a tutorial about this btw, but the link seems to be broken now)
d
Adding a transform plugin is just a shortcut to add a dbt package to packages.yml, so it depends on whether you want to store your models in a separate repo, or alongside the rest of your project inside 
transform/models
. When using a transform plugin, Meltano is able to automatically match the appropriate model to run in a given pipeline if the namespace of the extractor and some transform match, but any transforms in 
transform/models
are automatically run anyway
If you want to have more fine-tuned model selection logic, let me know 🙂
a
Gotcha! Just trying to figure out if I was coloring outside the lines.
But that’s pretty neat about the namespace matching.
what could also be nice is if Meltano had a way to autogenerate dbt sources from what it knows about the taps/targets used.
I think i saw an issue on the repo about that.
a
Yeah - this is something we’re excited about but it doesn’t exist yet today: Auto-generate dbt sources.yml from Extractor/Loader schema (#2266) · Issues · meltano / Meltano · GitLab
a
Neat. Yeah Im trying to think of the best way to approach that. The simplest initial thing is probably to just do codegen (ala https://hub.getdbt.com/fishtown-analytics/codegen/latest/)
It’s also interesting to consider a meltano specific source macro along the lines of 
{{ dbt_meltano.source('tap-name', 'stream') }}
. but i’m not sure what the macro’s entrypoint to the schema info would be.
oooh.. here’s an idea…. Meltano could possibly add the following to dbt;s 
packages.yml
:
Copy code
packages:
  - local: ../.meltano/dbt/something
and that package could be a continuously re-generated ‘sources’ package.
d
@andrew_stewart Ooh I like it! Wanna elaborate in the issue? 😄
a
Yeah
I wanted to type it out a bit to make sure it wasn’t a bad idea first
before it is imprinted immortal upon the pages of gitlab issues 😛
Open in Slack
PreviousNext
Powered by