https://linen.dev logo
Join Slack
Powered by
Next question for `dbt transforms` is there a way ...
# getting-started
j
Next question for 
dbt transforms
is there a way to invoke this from the command line without running the full EL aspect (which for stripe seems to be a full sync each time)?
t
For the current state, running 
--transform=only
is the way to go https://meltano.com/docs/command-line-interface.html#parameters-2 but, we’re working on a nice refactor which will make it much easier to do things like 
meltano run <transformation>
w/o all of the clutter
j
Thanks! I just found this. (No mention on the T page, so the command line --help options assisted.)
yes, 
meltano run transform
was what I was expecting. Not sure what multiple will look like yet for me!
t
expect to see more on this by the end of the month. https://gitlab.com/meltano/meltano/-/issues/2301
j
@taylor sorry to bother you with these n00b questions, but is there a quick way to prevent transform from installing the 
dbt-tap-stripe.git
on every run?
I want to do local dev on this
t
@john_romanski what commands are you running? Can you share some of the command line output?
j
Copy code
$ poetry run meltano elt tap-stripe target-postgres --transform only
meltano | Extract & load skipped.
meltano | Running transformation...
dbt     | Running with dbt=0.20.2
dbt     | Checking ../.meltano/transformers/dbt/target/*
dbt     | ERROR: not cleaning ../.meltano/transformers/dbt/target/* because it is protected
dbt     | Checking dbt_modules/*
dbt     |  Cleaned dbt_modules/*
dbt     | Checking logs/*
dbt     |  Cleaned logs/*
dbt     | Finished cleaning all paths.
dbt     | Running with dbt=0.20.2
dbt     | Installing <https://gitlab.com/DigitalTractor/dbt-tap-stripe.git@config-version-2>
dbt     |   Installed from revision config-version-2
dbt     | Running with dbt=0.20.2
dbt     | Found 12 models, 0 tests, 0 snapshots, 0 analyses, 148 macros, 0 operations, 0 seed files, 0 sources, 0 exposures
dbt     | The selection criterion 'my_meltano_project' does not match any nodes
this runs as expected, but it also runs the install every time (not desired, overwrites local changes)
t
oh, it’s running 
dbt deps
every time, I see
so, a couple of things here. since dbt is installed, if you navigate to the /transform directory you should just be able to use dbt natively
alternatively, you can invoke dbt, but the downside is you have to pass it more config/environment variables than you’d probably want.
j
I have this in a poetry project, which does not seem to be able to install/find dbt on it's own, how do I .meltano/transformers/dbt?
Copy code
/transform john$ ../.meltano/transformers/dbt/venv/bin/dbt run
Running with dbt=0.20.2
Encountered an error while reading the project:
  ERROR: Runtime Error
  Could not find profile named 'meltano'
Encountered an error:
Runtime Error
  Could not run dbt
transform john$ dbt
-bash: dbt: command not found
again sorry to bug you, I have been banging on this, and have some workarounds, but I don't know how to 
use dbt natively
from that dir
t
can you run 
poetry run meltano invoke dbt --models <your_model>
? I’d expect it to error out with a missing environment variable
we have https://gitlab.com/meltano/meltano/-/issues/2546 but this will probably be better solved with the 
environments
feature @edgar_ramirez_mondragon is working on
j
Copy code
poetry run meltano invoke dbt run --models stripe_customers
Running with dbt=0.20.2
Encountered an error:
Compilation Error
  Could not render {{ env_var('DBT_TARGET') }}: Env var required but not provided: 'DBT_TARGET'
You were right.
t
awesome - progress! 😄 since you’re using postgres, if you can set 
DBT_TARGET
to 
postgres
that should get what you’re looking for. I’m not sure the best way to do that with poetry though.
every once in a while I’ve got my head on straight 😅
I’m assuming you can set it in 
.env
or just export it via the command line
j
it does require all the vars as described 
DBT_TARGET=postgres
t
so you can update profiles.yml (being careful not to commit it) or export to the environment as well.
j
OK thanks! it seems like pipelines the UI is a list of/name for schedules in the meltano.yml? I would propose that pipeline could be a first class citizen, and the future transform would require that pipeline to provide the context (also called enviroments I guess).
meltano elt tap-stripe target-postgres --transform only
could be 
meltano transform stripe-pipeline
or 
meltano t stripe-pipeline
and you could see how 
meltano el stripe-pipeline
and 
meltano elt stripe-pipeline
would do different expected things
t
Sorry for the delay in replying! What you’re proposing is the path we’re on right now. This epic lays it out a bit more https://gitlab.com/groups/meltano/-/epics/116 Basically, we’re going to make meltano pipelines more composable and then update schedules into what you’re suggesting with pipelines. It’d all be under the 
meltano run
command and then you could do whatever you want 🙂
3 Views
Open in Slack
PreviousNext
Powered by