https://linen.dev logo
Join Slack
Powered by
Hi, i am looking for help with setting up dbt insi...
# announcements
f
Hi, i am looking for help with setting up dbt inside meltano. the docs and videos i found are all labeled as out of date. where do i go for help?
t
They’re out of date but they’re still mostly valid.
https://meltano.com/docs/transforms.html should still get you going
a
@finn_frotscher - I’m doing some dbt-in-meltano work right now (and had to troubleshoot some issues) if you have any specific questions.
f
thank you. i have a couple of conceptual questions. 1. to read the rich config of my loaders and extractors I can run 
m config target-postgres list
, but because of the naming convention for dbt projects i can not run 
m config tap-gitlab list
because this will always give me the loader config. correct?
2. with 
pip_url: <https://gitlab.com/meltano/dbt-tap-gitlab.git>
I can load a transform from an external git repo. but what if i want to define the transform locally? how would i load it and is there no way to name it other then to mirror the name of the respective extractor?
a
Im not sure I understand the first question.
For the 2nd question, you can indeed create new models (that aren’t from external dbt packages) directly under the dbt folder in your meltano project.
f
1. i have the 
extractor tap-gitlab
setup to 
loader target-postgres
Now i am struggling to get the 
transform tap-gitlab
working. i suspect there is an issue with the schema env variable. running 
$ meltano config tap-gitlab list
gives me the current config of 
extractor tap-gitlab
. how do i get config of 
transform tap-gitlab
example of 
$ meltano config tap-gitlab list
results in:
Copy code
api_url [env: TAP_GITLAB_API_URL] (default: '<https://gitlab.com>') current value: '<https://git.finnfrotscher.com>' (from `meltano.yml`)
	GitLab Instance: GitLab API/instance URL. When an API path is omitted, `/api/v4/` is assumed.
private_token [env: TAP_GITLAB_PRIVATE_TOKEN, GITLAB_API_TOKEN] (default: '') current value: 'xxx' (from `meltano.yml`: '$GITLAB_TOKEN')
	Access Token: GitLab personal access token or other API token.
groups [env: TAP_GITLAB_GROUPS, GITLAB_API_GROUPS] current value: '' (default)
	Groups: Space-separated names of groups to extract data from. Leave empty and provide a project name if you'd like to pull data from a project in a personal user namespace.
2. with dbt folder you mean the 
transform/
folder?
with 
custom_dbt
being the result of 
dbt init custom_dbt
?
a
Hmm, I just added models directly to dbt_project.yml and the models dir
You shouldn’t need to dbt init a second project
(wrt 1, im not as familiar with using meltano’s transform packages)
f
i think i got point 2. right now. thank you
but how do i call a locally defined model?
how does the transforms for a local model need to look like if i call it with like this?
MUST the name be my_meltano_project?
this is the error i get
i can run the model using 
cd transform && dbt run --profile-dir ./profile
t
@finn_frotscher what’s in the /transform directory is just a pure dbt project, so you’re right that any dbt commands work from there. You can also run 
meltano invoke dbt …
to run commands that way too.
f
an error is thrown when i use 
meltano invoke dbt run
but it works perfectly as expected when using 
dbt run
t
hrm. can you run with 
meltano --log-level=debug invoke …
Douwe also documented some other things you can do to override the behavior of meltano around dbt https://gitlab.com/meltano/meltano/-/issues/2629#note_523499492
f
```⟩ meltano --log-level=debug invoke dbt run ...ence/meltano/demo [2021-04-05 152112,787] [29112|MainThread|root] [DEBUG] Creating engine <meltano.core.project.Project object at 0x110f277f0>@sqlite:////Users/finnfrotscher/code/cofenster/business-intelligence/meltano/demo/.meltano/meltano.db [2021-04-05 152112,836] [29112|MainThread|urllib3.connectionpool] [DEBUG] Starting new HTTPS connection (1): www.meltano.com:443 [2021-04-05 152113,544] [29112|MainThread|urllib3.connectionpool] [DEBUG] https://www.meltano.com:443 "GET /discovery.yml HTTP/1.1" 200 91317 [2021-04-05 152114,494] [29112|MainThread|root] [WARNING] The 
discovery.yml
manifest received from https://www.meltano.com/discovery.yml has version 17, while this version of Meltano requires version 16. [2021-04-05 152114,494] [29112|MainThread|root] [WARNING] Please install the latest compatible version of Meltano using 
meltano upgrade
. [2021-04-05 152114,494] [29112|MainThread|root] [WARNING] Falling back on the cached 
discovery.yml
manifest... [2021-04-05 152114,699] [29112|MainThread|meltano.cli.utils] [DEBUG] argument type <class 'NoneType'> is not in the flattenalbe types (<class 'collections.abc.Mapping'>,) Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/meltano/cli/__init__.py", line 43, in main cli(obj={"project": None}) File "/usr/local/lib/python3.9/site-packages/click/core.py", line 829, in call return self.main(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/lib/python3.9/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/meltano/cli/params.py", line 23, in decorate return func(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/meltano/cli/params.py", line 57, in decorate func(project, *args, **kwargs) File "/usr/local/lib/python3.9/site-packages/meltano/cli/invoke.py", line 45, in invoke with invoker.prepared(session): File "/usr/local/Cellar/python@3.9/3.9.2_4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/contextlib.py", line 117, in enter return next(self.gen) File "/usr/local/lib/python3.9/site-packages/meltano/core/plugin_invoker.py", line 129, in prepared self.prepare(session) File "/usr/local/lib/python3.9/site-packages/meltano/core/plugin_invoker.py", line 102, in prepare self.plugin_config = self.settings_service.as_dict( File "/usr/local/lib/python3.9/site-packages/meltano/core/settings_service.py", line 134, in as_dict config_metadata = self.config_with_metadata(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/meltano/core/settings_service.py", line 113, in config_with_metadata for setting_def in self.definitions(extras=extras): File "/usr/local/lib/python3.9/site-packages/meltano/core/settings_service.py", line 341, in definitions for setting in self.setting_definitions File "/usr/local/lib/python3.9/site-packages/meltano/core/plugin/settings_service.py", line 68, in setting_definitions return self.plugin.settings_with_extras File "/usr/local/lib/python3.9/site-packages/meltano/core/plugin/project_plugin.py", line 187, in settings_with_extras return [*self.settings, *self.extra_settings] File "/usr/local/lib/python3.9/site-packages/meltano/core/plugin/project_plugin.py", line 174, in settings *SettingDefinition.from_missing(existing_settings, self.config), File …
t
I wonder if it’s expecting something after run 🤔
I almost never run just 
dbt run
I usually pass in something through 
--models
f
i will rm the view in my table and check if its created
nope. the view is not created when using 
meltano invoke dbt
but its created when using dbt directly
i have not understood models yet. i am trying to take it one by one
t
there might be a couple of errors conflating here. We don’t officially support python 3.9 yet, so that could be a problem. And also the latest dbt that’s installable with meltano is 16.1, though we’re updating that every soon.
f
which python version do you support and is there a suggested way of managing envs for meltano?
t
3.8 is the latest we support
using a 
.env
file is a good method
n
fwiw, there be dragons trying to run dbt via meltano invoke
t
oof yeah, good call out. We know the dbt integration could be greatly improved. a lot of that work is scheduled for May but obviously merge requests are very welcome 😄
n
but in any case
if you install the dbt transformer
and setup models for your taps
it all works nicely 🙂
i don’t quite remember the incantation, but I think 
meltano install transformer dbt
?
f
meltano add transformers dbt
i am now in a virtual with 
python --version
resulting in 
3.8.9
but the error persists. 
meltano | Transformation failed (1): hologram.ValidationError: Unable to create schema for 'Optional'
I think i am a step closer. after running 
meltano --log-level=debug invoke dbt run
again, i get this interesting bit:
Copy code
[DEBUG] Invoking: ['/Users/finnfrotscher/code/cofenster/business-intelligence/meltano/demo/.meltano/transformers/dbt/venv/bin/dbt', 'run']
[DEBUG] Env: { 
...
'DBT_TARGET_SCHEMA': 'analytics', 
'MELTANO_TRANSFORM_TARGET_SCHEMA': 'analytics', 
'DBT_MODELS': '  my_meltano_project', 
'MELTANO_TRANSFORM_MODELS': '  my_meltano_project'
'DBT_TARGET_SCHEMA': 'analytics', 
'MELTANO_TRANSFORM_TARGET_SCHEMA': 'analytics', 
'DBT_MODELS': '  my_meltano_project', 
'MELTANO_TRANSFORM_MODELS': '  my_meltano_project'
}
two oddities: 
'  my_meltano_project'
why the empty spaces? where does 
DBT_TARGET_SCHEMA: analytics
come from?
t
I think for 
my_meltano_project
it’s an artificat of how things are chained together. b/c you can pass multiple values to 
dbt run --models x y z
the space ensures things aren’t smushed together. but I’m not 100% sure
analytics
is the default schema defined in discovery.yml. It can be overridden
f
event when setting all the options:
i can confirm the env when running 
meltano --log-level=debug invoke dbt run
contains the settings:
Copy code
'DBT_TARGET_SCHEMA': 'api_testing',
'DBT_MODELS': 'my_meltano_project', 
'MELTANO_TRANSFORM_MODELS': 'my_meltano_project'
i still get the error `Unable to create schema for 'Optional'``
t
I don’t know where that error is coming from 😕 I’m poking around the codebase but it’s not obvious to me
I wonder if that’s a dbt thing? Are you confident the user on the database you’re connected to is able to create schemas etc?
I ask b/c we don’t have hologram in the Meltano project, but it seems like it’s used in dbt for JSON schema validation
but if dbt runs fine then something isn’t being passed correctly from Meltano to dbt
f
could it be because i am running an Apple M1 ?
t
no idea - that would be amazing though if that were the case
f
i notice it because i cant 
m add loaders target-postgres
since setting up the venv for python3.8
i get an install error in some gcc lib
t
oh wow - that’s interesting. Can you file an issue detailing your setup in https://gitlab.com/meltano/meltano/-/issues and I’ll see if I can get @aaronsteers to take a look? Douwe is on vacation this week and he knows more about the codebase than either of us so it may take a bit for him to get back to you unfortunately
a
@finn_frotscher to your question regarding M1, yes, that could be a part of the issue here. Another user reported issues with M1: https://gitlab.com/meltano/meltano/issues/2675
f
what infos should i include in the issue?
t
commands run, computer type, any other setup info would be helpful
f
ok will do. maybe include a 
meltano analytics
command. i allways find them helpfull
t
oh that’s a great idea!
f
t
@finn_frotscher can you take a look at the suggestion in this MR on how to fix that error? https://gitlab.com/meltano/meltano/-/merge_requests/2096/diffs
12 Views
Open in Slack
PreviousNext
Powered by