Hi, hopefully someone can help me out here. I am trying to use meltano to set up a data pipeline for us and the dbt transformation step keeps going a bit haywire. I set things up via docker and I can't seem to get things to play nice with the explore view for google analytics in the meltano UI. I did finally figure out that I needed to add a dbt_project.yml and profiles.yml to the transform folder to get the whole pipeline working from the UI. Should I just ignore the UI for now and only focus on the meltano CLI?

@nevin_morgan The CLI is definitely more stable than the UI right now, but the UI should still work, and we'd like for people to try it out and submit feedback so that we can make it live up to its potential 🙂 But I wouldn't suggest using it in production just yet.

I did finally figure out that I needed to add a dbt_project.yml and profiles.yml to the transform folder to get the whole pipeline working from the UI.

This is odd; those files should have been added to your project automatically as part of

meltano add transformer dbt

. Did you run that, or did you go straight to the UI, which may have resulted in that step being skipped?

I was in the UI and didn't see any documentation saying I needed to run

meltano add transformer dbt

so that is probably my issue. I thought running

meltano add

with

--include-related

would handle that for me. I also might be encountering some oddities because I am running things inside of docker since we will be deploying this to our kubernetes cluster.

You're right, having to explicitly add

transformer dbt

is implied by the sample code at https://meltano.com/#transformation but it's not explicitly documented... Like I mentioned in the thread just before yours, "Meltano's dbt/transformation support hasn't gotten a lot of love lately though, compared to its Singer support, so if this doesn't sound right or doesn't work as intended or expected, I'd love to work with you and anyone else on improving that 🙂"

I might be able to make some time for a call. I really want to sink my teeth into meltano first and get a grasp on it. It really seems to simplify running singer spec taps and targets which helps a lot. I am sort of feeling my way through this at the moment since we finally are embracing a proper data warehouse this is the first time I have had to sort out the whole data pipeline

@nevin_morgan Meltano should be able to handle it, it may just not be as smooth or well documented as the Singer side of things 🙂 So if you run into trouble, or can't get things to work that seem like they should be easy, please don't hesitate to ask here! You'll find I'm pretty responsive, with answers to questions as well as bug fixes 😉