https://meltano.com/ logo
Channels
Powered by
#announcements
Title
# announcements
m

most-monkey-23419

04/04/2021, 7:56 PM
Hi, i am looking for help with setting up dbt inside meltano. the docs and videos i found are all labeled as out of date. where do i go for help?
s

salmon-salesclerk-77709

04/04/2021, 7:56 PM
They’re out of date but they’re still mostly valid.
https://meltano.com/docs/transforms.html should still get you going
m

mammoth-napkin-71897

04/04/2021, 8:11 PM
@most-monkey-23419 - I’m doing some dbt-in-meltano work right now (and had to troubleshoot some issues) if you have any specific questions.
m

most-monkey-23419

04/04/2021, 8:16 PM
thank you. i have a couple of conceptual questions. 1. to read the rich config of my loaders and extractors I can run 
m config target-postgres list
, but because of the naming convention for dbt projects i can not run 
m config tap-gitlab list
because this will always give me the loader config. correct?
2. with 
pip_url: <https://gitlab.com/meltano/dbt-tap-gitlab.git>
I can load a transform from an external git repo. but what if i want to define the transform locally? how would i load it and is there no way to name it other then to mirror the name of the respective extractor?
m

mammoth-napkin-71897

04/04/2021, 8:37 PM
Im not sure I understand the first question.
For the 2nd question, you can indeed create new models (that aren’t from external dbt packages) directly under the dbt folder in your meltano project.
m

most-monkey-23419

04/04/2021, 8:52 PM
1. i have the 
extractor tap-gitlab
setup to 
loader target-postgres
Now i am struggling to get the 
transform tap-gitlab
working. i suspect there is an issue with the schema env variable. running 
$ meltano config tap-gitlab list
gives me the current config of 
extractor tap-gitlab
. how do i get config of 
transform tap-gitlab
example of 
$ meltano config tap-gitlab list
results in:
Copy code
api_url [env: TAP_GITLAB_API_URL] (default: '<https://gitlab.com>') current value: '<https://git.finnfrotscher.com>' (from `meltano.yml`)
	GitLab Instance: GitLab API/instance URL. When an API path is omitted, `/api/v4/` is assumed.
private_token [env: TAP_GITLAB_PRIVATE_TOKEN, GITLAB_API_TOKEN] (default: '') current value: 'xxx' (from `meltano.yml`: '$GITLAB_TOKEN')
	Access Token: GitLab personal access token or other API token.
groups [env: TAP_GITLAB_GROUPS, GITLAB_API_GROUPS] current value: '' (default)
	Groups: Space-separated names of groups to extract data from. Leave empty and provide a project name if you'd like to pull data from a project in a personal user namespace.
2. with dbt folder you mean the 
transform/
folder?
with 
custom_dbt
being the result of 
dbt init custom_dbt
?
m

mammoth-napkin-71897

04/04/2021, 10:11 PM
Hmm, I just added models directly to dbt_project.yml and the models dir
You shouldn’t need to dbt init a second project
🙌 1
(wrt 1, im not as familiar with using meltano’s transform packages)
m

most-monkey-23419

04/05/2021, 11:33 AM
i think i got point 2. right now. thank you
but how do i call a locally defined model?
how does the transforms for a local model need to look like if i call it with like this?
MUST the name be my_meltano_project?
this is the error i get
i can run the model using 
cd transform && dbt run --profile-dir ./profile
s

salmon-salesclerk-77709

04/05/2021, 1:17 PM
@most-monkey-23419 what’s in the /transform directory is just a pure dbt project, so you’re right that any dbt commands work from there. You can also run 
meltano invoke dbt …
to run commands that way too.
m

most-monkey-23419

04/05/2021, 1:19 PM
an error is thrown when i use 
meltano invoke dbt run
but it works perfectly as expected when using 
dbt run
melty bouncy 1
s

salmon-salesclerk-77709

04/05/2021, 1:20 PM
hrm. can you run with 
meltano --log-level=debug invoke …
Douwe also documented some other things you can do to override the behavior of meltano around dbt https://gitlab.com/meltano/meltano/-/issues/2629#note_523499492
m

most-monkey-23419

04/05/2021, 1:21 PM
Copy code
⟩ meltano --log-level=debug invoke dbt run                                                                                                                    ...ence/meltano/demo
[2021-04-05 15:21:12,787] [29112|MainThread|root] [DEBUG] Creating engine <meltano.core.project.Project object at 0x110f277f0>@sqlite:////Users/finnfrotscher/code/cofenster/business-intelligence/meltano/demo/.meltano/meltano.db
[2021-04-05 15:21:12,836] [29112|MainThread|urllib3.connectionpool] [DEBUG] Starting new HTTPS connection (1): <http://www.meltano.com:443|www.meltano.com:443>
[2021-04-05 15:21:13,544] [29112|MainThread|urllib3.connectionpool] [DEBUG] <https://www.meltano.com:443> "GET /discovery.yml HTTP/1.1" 200 91317
[2021-04-05 15:21:14,494] [29112|MainThread|root] [WARNING] The `discovery.yml` manifest received from <https://www.meltano.com/discovery.yml> has version 17, while this version of Meltano requires version 16.
[2021-04-05 15:21:14,494] [29112|MainThread|root] [WARNING] Please install the latest compatible version of Meltano using `meltano upgrade`.
[2021-04-05 15:21:14,494] [29112|MainThread|root] [WARNING] Falling back on the cached `discovery.yml` manifest...
[2021-04-05 15:21:14,699] [29112|MainThread|meltano.cli.utils] [DEBUG] argument type <class 'NoneType'> is not in the flattenalbe types (<class 'collections.abc.Mapping'>,)
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/meltano/cli/__init__.py", line 43, in main
    cli(obj={"project": None})
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/meltano/cli/params.py", line 23, in decorate
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/meltano/cli/params.py", line 57, in decorate
    func(project, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/meltano/cli/invoke.py", line 45, in invoke
    with invoker.prepared(session):
  File "/usr/local/Cellar/python@3.9/3.9.2_4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/contextlib.py", line 117, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.9/site-packages/meltano/core/plugin_invoker.py", line 129, in prepared
    self.prepare(session)
  File "/usr/local/lib/python3.9/site-packages/meltano/core/plugin_invoker.py", line 102, in prepare
    self.plugin_config = self.settings_service.as_dict(
  File "/usr/local/lib/python3.9/site-packages/meltano/core/settings_service.py", line 134, in as_dict
    config_metadata = self.config_with_metadata(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/meltano/core/settings_service.py", line 113, in config_with_metadata
    for setting_def in self.definitions(extras=extras):
  File "/usr/local/lib/python3.9/site-packages/meltano/core/settings_service.py", line 341, in definitions
    for setting in self.setting_definitions
  File "/usr/local/lib/python3.9/site-packages/meltano/core/plugin/settings_service.py", line 68, in setting_definitions
    return self.plugin.settings_with_extras
  File "/usr/local/lib/python3.9/site-packages/meltano/core/plugin/project_plugin.py", line 187, in settings_with_extras
    return [*self.settings, *self.extra_settings]
  File "/usr/local/lib/python3.9/site-packages/meltano/core/plugin/project_plugin.py", line 174, in settings
    *SettingDefinition.from_missing(existing_settings, self.config),
  File "/usr/local/lib/python3.9/site-packages/meltano/core/setting_definition.py", line 98, in from_missing
    flat_config = flatten(config, "dot")
  File "/usr/local/lib/python3.9/site-packages/meltano/core/utils/__init__.py", line 179, in flatten
    return flatten_dict.flatten(d, reducer, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/flatten_dict/flatten_dict.py", line 46, in flatten
    raise ValueError("argument type %s is not in the flattenalbe types %s"
ValueError: argument type <class 'NoneType'> is not in the flattenalbe types (<class 'collections.abc.Mapping'>,)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/meltano/cli/__init__.py", line 51, in main
    raise CliError(str(err)) from err
meltano.cli.utils.CliError: argument type <class 'NoneType'> is not in the flattenalbe types (<class 'collections.abc.Mapping'>,)
argument type <class 'NoneType'> is not in the flattenalbe types (<class 'collections.abc.Mapping'>,)
s

salmon-salesclerk-77709

04/05/2021, 1:22 PM
I wonder if it’s expecting something after run 🤔
I almost never run just 
dbt run
I usually pass in something through 
--models
m

most-monkey-23419

04/05/2021, 1:23 PM
i will rm the view in my table and check if its created
nope. the view is not created when using 
meltano invoke dbt
but its created when using dbt directly
i have not understood models yet. i am trying to take it one by one
s

salmon-salesclerk-77709

04/05/2021, 1:26 PM
there might be a couple of errors conflating here. We don’t officially support python 3.9 yet, so that could be a problem. And also the latest dbt that’s installable with meltano is 16.1, though we’re updating that every soon.
m

most-monkey-23419

04/05/2021, 1:28 PM
which python version do you support and is there a suggested way of managing envs for meltano?
s

salmon-salesclerk-77709

04/05/2021, 1:28 PM
3.8 is the latest we support
using a 
.env
file is a good method
b

billowy-barista-37174

04/05/2021, 1:41 PM
fwiw, there be dragons trying to run dbt via meltano invoke
s

salmon-salesclerk-77709

04/05/2021, 1:43 PM
oof yeah, good call out. We know the dbt integration could be greatly improved. a lot of that work is scheduled for May but obviously merge requests are very welcome 😄
b

billowy-barista-37174

04/05/2021, 1:44 PM
but in any case
if you install the dbt transformer
and setup models for your taps
it all works nicely 🙂
i don’t quite remember the incantation, but I think 
meltano install transformer dbt
?
m

most-monkey-23419

04/05/2021, 1:51 PM
meltano add transformers dbt
i am now in a virtual with 
python --version
resulting in 
3.8.9
but the error persists. 
meltano | Transformation failed (1): hologram.ValidationError: Unable to create schema for 'Optional'
I think i am a step closer. after running 
meltano --log-level=debug invoke dbt run
again, i get this interesting bit:
Copy code
[DEBUG] Invoking: ['/Users/finnfrotscher/code/cofenster/business-intelligence/meltano/demo/.meltano/transformers/dbt/venv/bin/dbt', 'run']
[DEBUG] Env: { 
...
'DBT_TARGET_SCHEMA': 'analytics', 
'MELTANO_TRANSFORM_TARGET_SCHEMA': 'analytics', 
'DBT_MODELS': '  my_meltano_project', 
'MELTANO_TRANSFORM_MODELS': '  my_meltano_project'
'DBT_TARGET_SCHEMA': 'analytics', 
'MELTANO_TRANSFORM_TARGET_SCHEMA': 'analytics', 
'DBT_MODELS': '  my_meltano_project', 
'MELTANO_TRANSFORM_MODELS': '  my_meltano_project'
}
two oddities: 
'  my_meltano_project'
why the empty spaces? where does 
DBT_TARGET_SCHEMA: analytics
come from?
s

salmon-salesclerk-77709

04/05/2021, 2:05 PM
I think for 
my_meltano_project
it’s an artificat of how things are chained together. b/c you can pass multiple values to 
dbt run --models x y z
the space ensures things aren’t smushed together. but I’m not 100% sure
analytics
is the default schema defined in discovery.yml. It can be overridden
m

most-monkey-23419

04/05/2021, 2:24 PM
event when setting all the options:
i can confirm the env when running 
meltano --log-level=debug invoke dbt run
contains the settings:
Copy code
'DBT_TARGET_SCHEMA': 'api_testing',
'DBT_MODELS': 'my_meltano_project', 
'MELTANO_TRANSFORM_MODELS': 'my_meltano_project'
i still get the error `Unable to create schema for 'Optional'``
s

salmon-salesclerk-77709

04/05/2021, 2:30 PM
I don’t know where that error is coming from 😕 I’m poking around the codebase but it’s not obvious to me
I wonder if that’s a dbt thing? Are you confident the user on the database you’re connected to is able to create schemas etc?
I ask b/c we don’t have hologram in the Meltano project, but it seems like it’s used in dbt for JSON schema validation
but if dbt runs fine then something isn’t being passed correctly from Meltano to dbt
m

most-monkey-23419

04/05/2021, 2:45 PM
could it be because i am running an Apple M1 ?
s

salmon-salesclerk-77709

04/05/2021, 2:45 PM
no idea - that would be amazing though if that were the case
m

most-monkey-23419

04/05/2021, 2:46 PM
i notice it because i cant 
m add loaders target-postgres
since setting up the venv for python3.8
i get an install error in some gcc lib
s

salmon-salesclerk-77709

04/05/2021, 2:48 PM
oh wow - that’s interesting. Can you file an issue detailing your setup in https://gitlab.com/meltano/meltano/-/issues and I’ll see if I can get @salmon-actor-23953 to take a look? Douwe is on vacation this week and he knows more about the codebase than either of us so it may take a bit for him to get back to you unfortunately
s

salmon-actor-23953

04/05/2021, 3:40 PM
@most-monkey-23419 to your question regarding M1, yes, that could be a part of the issue here. Another user reported issues with M1: https://gitlab.com/meltano/meltano/issues/2675
m

most-monkey-23419

04/05/2021, 4:08 PM
what infos should i include in the issue?
s

salmon-salesclerk-77709

04/05/2021, 4:09 PM
commands run, computer type, any other setup info would be helpful
m

most-monkey-23419

04/05/2021, 4:20 PM
ok will do. maybe include a 
meltano analytics
command. i allways find them helpfull
s

salmon-salesclerk-77709

04/05/2021, 4:53 PM
oh that’s a great idea!
m

most-monkey-23419

04/13/2021, 9:51 AM
s

salmon-salesclerk-77709

04/13/2021, 3:10 PM
@most-monkey-23419 can you take a look at the suggestion in this MR on how to fix that error? https://gitlab.com/meltano/meltano/-/merge_requests/2096/diffs
Powered by