https://linen.dev logo
Join Slack
Powered by
Hey everyone! If I want to configure a tap to use ...
# announcements
r
Hey everyone! If I want to configure a tap to use a custom dbt transform in the 
transform
directory, what changes will be reflected in my 
meltano.yml
? I can't seem to find the documentation on this. The ones I've found are showing what to do if I am importing a dbt package, but not sure how to set a custom transform to a particular tap. Thank you!
v
My understanding is that meltano runs 
dbt run
so you just configure whatever you want in the transform folder
r
Thanks for the reploy @visch. I get that but I know I need to configure something in the 
meltano.yml
file. However, all of the examples I am looking at load in dbt packages and I don't see how to utilize a custom transform I have defined in the 
transform/
directory
Following up on this^ still cannot get meltano to run a dbt custom transform. I'm trying to use a dbt project defined in a private Git repository and I'm getting 
Transform <transform_name> is not known to Meltano
. Anybody been through this before?
v
@taylor ? I'll bow out for a bit as I haven't dove too deep with dbt and Meltano yet!
r
Also @eric_simmerman - have you ever set up a Meltano project with your 
tap-spreadsheets-anywhere
tap that also runs custom dbt transform code?
e
negative - although the tap should be immaterial since it writes Singer spec records at the end of the day - just like every other tap.
t
so full disclosure for me I’ve yet to run a custom dbt transformation through meltano. That said, it’s a full-fledged dbt project w/in the /transform folder so you should be able to navigate to that directory and run dbt w/o going through Meltano. https://meltano.com/tutorials/create-custom-transforms-and-models.html#adding-custom-transforms doesn’t seem to provide much guidance either.
This is one thing we definitely want to work on but has been deprioritized when Douwe was focusing on just EL
r
Got it @taylor. So there's no clear path forward on how to configure a custom dbt transform with a meltano pipeline? If that were the case, would we just have to orchestrate dbt to run after the meltano EL pipeline? I'd love to have them connected, so if there is any guidance on how we can do this, I'd be happy to help out with documentation after the fact
t
@douwe_maan can you validate what I’ve said here regarding dbt?
r
Or even an example project would be great to sift through just to understand how this would work
t
@ricky_renner just remembered Douwe left a comment https://gitlab.com/meltano/meltano/-/issues/2629#note_523499492 about the dbt integration too
r
Thank you @taylor for that thread that was really helpful! I'm now getting:
Copy code
Your Meltano project does not contain a transform plugin for this extractor. Only proceed with running transformations as part of your pipeline if you've added these manually.
I know I need to add a transform plugin for the 
tap-spreadsheets-anywhere
tap but it looks like I can only add custom plugins for extractors and loaders: https://meltano.com/docs/plugin-management.html#custom-plugins
t
@ricky_renner that seems expected. It’s basically saying there’s no default set of transforms for that tap (which to me makes sense b/c a spreadsheet can be anything) so you’ll have a custom set of transforms based on whatever data your inputting
r
@taylor that makes sense. Apologies if this is an obvious question, but I assume this would mean that this is how my 
meltano.yml
file would have to be configured:
Copy code
# etc
plugins:
  extractors:
  - name: tap-spreadsheets-anywhere
    pip_url: git+<https://github.com/ets/tap-spreadsheets-anywhere.git>
    config:
      tables:
      - path: <gs://bucket-name>
        name: example
        pattern: blob-path/folder/.*
        start_date: '2017-05-01T00:00:00Z'
        delimiter: '|'
        key_properties:
        - id
        format: csv
# etc
  transforms:
  - name: tap-spreadsheets-anywhere
    # other settings here...
But I just don't really see how I add a custom dbt package to my project so that I could have a meltano transform reference it. Here is some of the documentation I am looking at: https://meltano.com/tutorials/create-custom-transforms-and-models.html#prerequisites I'm trying to accomplish this task of adding a custom dbt package to my project in two different ways and am failing at both: 1. Add an entire DBT project underneath the 
transform
directory. If I do this, how would I configure my 
transforms:
plugin section in my 
meltano.yml
? 2. Add a section in my 
meltano.yml
that downloads the dbt package via a reference to the GitHub repo.
t
@ricky_renner based on Douwe’s comment I think you could do:
Copy code
# etc
plugins:
  extractors:
  - name: tap-spreadsheets-anywhere
    pip_url: git+<https://github.com/ets/tap-spreadsheets-anywhere.git>
    model_name: my_dbt_models # this is where you can put whatever dbt models you want
    config:
      tables:
      - path: <gs://bucket-name>
        name: example
        pattern: blob-path/folder/.*
        start_date: '2017-05-01T00:00:00Z'
        delimiter: '|'
        key_properties:
        - id
        format: csv
# etc
  transformers:
  - name: dbt    # other settings here...
    config:
      models: $MELTANO_EXTRACT__MODEL_NAME
and then you could run the elt with 
--transform=only
argh hit enter before I edited
one sec
r
I also did the same thing haha
I had those changes in, but what you have right there is actually what would go under the 
transformers
plugin section pretty sure
From what I am reading in the documentation, it sounds like there is a 
transforms
section and a 
transformers
section. Where 
transformers
is where something like 
dbt
is defined and configured, but 
transforms
is where dbt packages would be defined and configured
t
oh you’re right. yeah if you update my example to be 
transformers
then that would override what’s passed to meltano elt
r
But what it seems like is I also need to define something in the 
transforms
plugin section, but I don't really understand how to add a custom dbt package to that section. Does that make sense?
t
Is it actually a dbt package or just code in `/transform`directory? The link you just shared points to full dbt packages which are a separate thing.
d
@ricky_renner @taylor Wanna jump on Zoom for moment?
r
@douwe_maan @taylor that would be great!
d
6 Views
Open in Slack
PreviousNext
Powered by