https://meltano.com/ logo
Channels
Powered by
#announcements
Title
# announcements
p

proud-pillow-55935

03/18/2021, 9:56 PM
Hey everyone! If I want to configure a tap to use a custom dbt transform in the 
transform
directory, what changes will be reflected in my 
meltano.yml
? I can't seem to find the documentation on this. The ones I've found are showing what to do if I am importing a dbt package, but not sure how to set a custom transform to a particular tap. Thank you!
f

flat-bear-81546

03/19/2021, 1:10 AM
My understanding is that meltano runs 
dbt run
so you just configure whatever you want in the transform folder
p

proud-pillow-55935

03/19/2021, 1:24 PM
Thanks for the reploy @flat-bear-81546. I get that but I know I need to configure something in the 
meltano.yml
file. However, all of the examples I am looking at load in dbt packages and I don't see how to utilize a custom transform I have defined in the 
transform/
directory
Following up on this^ still cannot get meltano to run a dbt custom transform. I'm trying to use a dbt project defined in a private Git repository and I'm getting 
Transform <transform_name> is not known to Meltano
. Anybody been through this before?
f

flat-bear-81546

03/22/2021, 2:16 PM
@salmon-salesclerk-77709 ? I'll bow out for a bit as I haven't dove too deep with dbt and Meltano yet!
p

proud-pillow-55935

03/22/2021, 2:24 PM
Also @miniature-answer-26495 - have you ever set up a Meltano project with your 
tap-spreadsheets-anywhere
tap that also runs custom dbt transform code?
m

miniature-answer-26495

03/22/2021, 2:27 PM
negative - although the tap should be immaterial since it writes Singer spec records at the end of the day - just like every other tap.
s

salmon-salesclerk-77709

03/22/2021, 2:32 PM
so full disclosure for me I’ve yet to run a custom dbt transformation through meltano. That said, it’s a full-fledged dbt project w/in the /transform folder so you should be able to navigate to that directory and run dbt w/o going through Meltano. https://meltano.com/tutorials/create-custom-transforms-and-models.html#adding-custom-transforms doesn’t seem to provide much guidance either.
This is one thing we definitely want to work on but has been deprioritized when Douwe was focusing on just EL
😔 1
p

proud-pillow-55935

03/22/2021, 2:55 PM
Got it @salmon-salesclerk-77709. So there's no clear path forward on how to configure a custom dbt transform with a meltano pipeline? If that were the case, would we just have to orchestrate dbt to run after the meltano EL pipeline? I'd love to have them connected, so if there is any guidance on how we can do this, I'd be happy to help out with documentation after the fact
s

salmon-salesclerk-77709

03/22/2021, 2:58 PM
@ripe-musician-59933 can you validate what I’ve said here regarding dbt?
p

proud-pillow-55935

03/22/2021, 3:04 PM
Or even an example project would be great to sift through just to understand how this would work
s

salmon-salesclerk-77709

03/22/2021, 3:36 PM
@proud-pillow-55935 just remembered Douwe left a comment https://gitlab.com/meltano/meltano/-/issues/2629#note_523499492 about the dbt integration too
🙌 1
p

proud-pillow-55935

03/22/2021, 4:57 PM
Thank you @salmon-salesclerk-77709 for that thread that was really helpful! I'm now getting:
Copy code
Your Meltano project does not contain a transform plugin for this extractor. Only proceed with running transformations as part of your pipeline if you've added these manually.
I know I need to add a transform plugin for the 
tap-spreadsheets-anywhere
tap but it looks like I can only add custom plugins for extractors and loaders: https://meltano.com/docs/plugin-management.html#custom-plugins
s

salmon-salesclerk-77709

03/22/2021, 5:26 PM
@proud-pillow-55935 that seems expected. It’s basically saying there’s no default set of transforms for that tap (which to me makes sense b/c a spreadsheet can be anything) so you’ll have a custom set of transforms based on whatever data your inputting
p

proud-pillow-55935

03/22/2021, 5:31 PM
@salmon-salesclerk-77709 that makes sense. Apologies if this is an obvious question, but I assume this would mean that this is how my 
meltano.yml
file would have to be configured:
Copy code
# etc
plugins:
  extractors:
  - name: tap-spreadsheets-anywhere
    pip_url: git+<https://github.com/ets/tap-spreadsheets-anywhere.git>
    config:
      tables:
      - path: <gs://bucket-name>
        name: example
        pattern: blob-path/folder/.*
        start_date: '2017-05-01T00:00:00Z'
        delimiter: '|'
        key_properties:
        - id
        format: csv
# etc
  transforms:
  - name: tap-spreadsheets-anywhere
    # other settings here...
But I just don't really see how I add a custom dbt package to my project so that I could have a meltano transform reference it. Here is some of the documentation I am looking at: https://meltano.com/tutorials/create-custom-transforms-and-models.html#prerequisites I'm trying to accomplish this task of adding a custom dbt package to my project in two different ways and am failing at both: 1. Add an entire DBT project underneath the 
transform
directory. If I do this, how would I configure my 
transforms:
plugin section in my 
meltano.yml
? 2. Add a section in my 
meltano.yml
that downloads the dbt package via a reference to the GitHub repo.
s

salmon-salesclerk-77709

03/22/2021, 5:40 PM
@proud-pillow-55935 based on Douwe’s comment I think you could do:
Copy code
# etc
plugins:
  extractors:
  - name: tap-spreadsheets-anywhere
    pip_url: git+<https://github.com/ets/tap-spreadsheets-anywhere.git>
    model_name: my_dbt_models # this is where you can put whatever dbt models you want
    config:
      tables:
      - path: <gs://bucket-name>
        name: example
        pattern: blob-path/folder/.*
        start_date: '2017-05-01T00:00:00Z'
        delimiter: '|'
        key_properties:
        - id
        format: csv
# etc
  transformers:
  - name: dbt    # other settings here...
    config:
      models: $MELTANO_EXTRACT__MODEL_NAME
and then you could run the elt with 
--transform=only
argh hit enter before I edited
one sec
p

proud-pillow-55935

03/22/2021, 5:42 PM
I also did the same thing haha
I had those changes in, but what you have right there is actually what would go under the 
transformers
plugin section pretty sure
From what I am reading in the documentation, it sounds like there is a 
transforms
section and a 
transformers
section. Where 
transformers
is where something like 
dbt
is defined and configured, but 
transforms
is where dbt packages would be defined and configured
s

salmon-salesclerk-77709

03/22/2021, 5:45 PM
oh you’re right. yeah if you update my example to be 
transformers
then that would override what’s passed to meltano elt
p

proud-pillow-55935

03/22/2021, 5:46 PM
But what it seems like is I also need to define something in the 
transforms
plugin section, but I don't really understand how to add a custom dbt package to that section. Does that make sense?
s

salmon-salesclerk-77709

03/22/2021, 5:50 PM
Is it actually a dbt package or just code in `/transform`directory? The link you just shared points to full dbt packages which are a separate thing.
r

ripe-musician-59933

03/22/2021, 5:52 PM
@proud-pillow-55935 @salmon-salesclerk-77709 Wanna jump on Zoom for moment?
👍 1
p

proud-pillow-55935

03/22/2021, 5:53 PM
@ripe-musician-59933 @salmon-salesclerk-77709 that would be great!
r

ripe-musician-59933

03/22/2021, 5:53 PM
Powered by