https://linen.dev logo
Join Slack
Powered by
I am in the midst of trying to upgrade/revive my m...
# troubleshooting
e
I am in the midst of trying to upgrade/revive my meltano project... installed latest 
v2.4.0
with a newer airflow etc... but DBT seems to still be complaining.. I went ahead and deleted 
transform/profile
... but still seeing an error when I go to execute my project (See thread)
Copy code
2022-08-13T17:00:53.523575Z [error    ] Plugin 'Transformer 'dbt' not found.
Use of the legacy 'dbt' Transformer is deprecated in favor of new adapter specific implementations (e.g. 'dbt-snowflake') compatible with the 'meltano run ...' command.
<https://docs.meltano.com/guide/transformation>
To continue using the legacy 'dbt' Transformer, add it to your Project using 'meltano add transformer dbt'.' is not known to Meltano 
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/meltano/core/project_plugins_service.py", line 303, in get_plugin
    plugin = next(
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/meltano/core/elt_context.py", line 409, in plugin_context
    plugin = self.plugins_service.get_plugin(plugin_ref)
  File "/usr/local/lib/python3.10/dist-packages/meltano/core/project_plugins_service.py", line 311, in get_plugin
    raise PluginNotFoundError(plugin_ref) from stop
meltano.core.plugin.error.PluginNotFoundError: Transformer 'dbt' is not known to Meltano

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/meltano/core/logging/output_logger.py", line 138, in redirect_logging
    yield
  File "/usr/local/lib/python3.10/dist-packages/meltano/cli/elt.py", line 259, in _redirect_output
    yield
  File "/usr/local/lib/python3.10/dist-packages/meltano/cli/elt.py", line 273, in _run_elt
    elt_context = context_builder.context()
  File "/usr/local/lib/python3.10/dist-packages/meltano/core/elt_context.py", line 480, in context
    transformer = self.plugin_context(
  File "/usr/local/lib/python3.10/dist-packages/meltano/core/elt_context.py", line 412, in plugin_context
    raise PluginNotFoundError(
meltano.core.plugin.error.PluginNotFoundError: Plugin 'Transformer 'dbt' not found.
Use of the legacy 'dbt' Transformer is deprecated in favor of new adapter specific implementations (e.g. 'dbt-snowflake') compatible with the 'meltano run ...' command.
<https://docs.meltano.com/guide/transformation>
To continue using the legacy 'dbt' Transformer, add it to your Project using 'meltano add transformer dbt'.' is not known to Meltano
i've redone 
meltano remove transformer dbt
and 
meltano remove files dbt
... no change
a
Hi, @emcp! Which snowflake adapter are you wanting to use? Have you tried running 
meltano add transformer dbt-snowflake
or similar?
More info on that process is at the link referenced in the log: https://docs.meltano.com/guide/transformation
Sharing here in case you've not already found this, our 2.0 migration guide may also be helpful: https://docs.meltano.com/guide/v2-migration
e
Hi AJ, long time
yeah I am on the migration page..
but that first link is new to me, will make dinner and check it OUT.. thank you
I just went and refreshed the meltano juju charm so it can rock the new 2.4.0
a
Great! Will look forward to hearing how it goes!
Also open to suggestions on how we can improve the docs and/or make them easier to discover. 😁
e
Copy code
$ meltano add transformer dbt-postgres
Transformer 'dbt-postgres' already exists in your Meltano project
To add it to your project another time so that each can be configured differently,
add a new plugin inheriting from the existing one with its own unique name:
        meltano add transformer dbt-postgres--new --inherit-from dbt-postgres

Lockfile already exists: /home/ubuntu/meltano_proj_repo/plugins/files/files-dbt-postgres--meltano.lock

$ ls -la
total 9
drwxrwxr-x  4 ubuntu ubuntu   9 Aug 13 19:09  .
drwxrwxr-x 12 ubuntu ubuntu  18 Aug 13 19:00  ..
-rwxrwxr-x  1 ubuntu ubuntu  26 Aug 13 18:53  .gitignore
-rwxrwxr-x  1 ubuntu ubuntu  27 Aug 13 18:53 '.gitignore (files-dbt-postgres)'
-rwxrwxr-x  1 ubuntu ubuntu   0 Aug 13 18:53  .gitkeep
-rwxrwxr-x  1 ubuntu ubuntu 466 Aug 13 18:53 'dbt_project (files-dbt-postgres).yml'
-rwxrwxr-x  1 ubuntu ubuntu 466 Aug 13 18:53  dbt_project.yml
drwxrwxr-x  3 ubuntu ubuntu   5 Aug 13 18:53  models
drwxrwxr-x  3 ubuntu ubuntu   3 Aug 13 18:53  profiles
ubuntu@juju-45fbf0-38:~/meltano_proj_repo/transform$
hmmm I'm feeling I might just wipe the transformers folder entirely and try from zero here.. when I did the migration and followed the instruction it didn't mention these other doubled up files.. I am guessing I erase one or the other
ahhh found it
Copy code
To complete your migration, consolidate dbt_project.yml and dbt_project (<adapter name>).yml into a single file called dbt_project.yml
i wish if it could have seen the two files match and just pick one
seems I have no changes in mine
my meltano.yml looks updated but.. seems when I invoke via the GUI it's somehow calling out for old dbt.. here's what i have in the yml
Copy code
transformers:
  - name: dbt-postgres
    variant: dbt-labs
    pip_url: dbt-core~=1.1.0 dbt-postgres~=1.1.0
gonna keep poking around
ah.. i think I found the problem
in the 
dbt_project
under 
target-path
i am seeing
Copy code
snapshot-paths:
- snapshots
target-path: ../.meltano/transformers/dbt/target
log-path: logs
perhaps I will manually try renaming this to .. "dbt-postgres"
i just re-checked.. I did the migration (I think) correctly.. and saw new files generated.. but the targets seem to still be pointing to the old dbt..
voila.. yes... I think the migration tool/routine/process didn't update the path (or I hope I didn't skip something)
hmm sadly even if I manually edit the path it failed with same error
I will keep trying.. I had yet to perform the config step to the project .. as outlined in the docs so I will make dinner and come back and try that
Copy code
sudo meltano invoke dbt-postgres run
18:04:44  Encountered an error while reading profiles:
18:04:44    ERROR: Runtime Error
  The profile 'meltano' does not have a target named ''. The valid target names for this profile are:
   - dev
   - staging
   - prod
18:04:44  Defined profiles:
18:04:44   - meltano
18:04:44  
For more information on configuring profiles, please consult the dbt docs:

<https://docs.getdbt.com/docs/configure-your-profile>

18:04:44  Encountered an error:
Runtime Error
  Could not run dbt
Just tried to add a new pipeline.. thinking maaybe that part of the YAML is still old and from the prior setup.. I think I need to back track and try pipx now within the container because it just seems like installing straight into the container is leading to things not aligning as expected
when I create the pipline without transformations selected.. at least I get a new error
Copy code
2022-08-13T20:55:00.995574Z [info     ] Running extract & load...      name=meltano run_id=67ec74b6-9c9d-4a22-a6bc-10f2655b0d72 state_id=ibkr-tickers-to-postgres-3
2022-08-13T20:55:01.294070Z [info     ] Traceback (most recent call last): cmd_type=extractor name=tap-ibkr-tickers run_id=67ec74b6-9c9d-4a22-a6bc-10f2655b0d72 state_id=ibkr-tickers-to-postgres-3 stdio=stderr
2022-08-13T20:55:01.294306Z [info     ]   File "/usr/local/bin/poetry", line 5, in <module> cmd_type=extractor name=tap-ibkr-tickers run_id=67ec74b6-9c9d-4a22-a6bc-10f2655b0d72 state_id=ibkr-tickers-to-postgres-3 stdio=stderr
2022-08-13T20:55:01.294429Z [info     ]     from poetry import console cmd_type=extractor name=tap-ibkr-tickers run_id=67ec74b6-9c9d-4a22-a6bc-10f2655b0d72 state_id=ibkr-tickers-to-postgres-3 stdio=stderr
2022-08-13T20:55:01.294538Z [info     ]   File "/usr/local/lib/python3.10/dist-packages/poetry/__init__.py", line 1, in <module> cmd_type=extractor name=tap-ibkr-tickers run_id=67ec74b6-9c9d-4a22-a6bc-10f2655b0d72 state_id=ibkr-tickers-to-postgres-3 stdio=stderr
2022-08-13T20:55:01.294673Z [info     ]     from poetry.console import Application cmd_type=extractor name=tap-ibkr-tickers run_id=67ec74b6-9c9d-4a22-a6bc-10f2655b0d72 state_id=ibkr-tickers-to-postgres-3 stdio=stderr
2022-08-13T20:55:01.294784Z [info     ]   File "/usr/local/lib/python3.10/dist-packages/poetry/console/__init__.py", line 1, in <module> cmd_type=extractor name=tap-ibkr-tickers run_id=67ec74b6-9c9d-4a22-a6bc-10f2655b0d72 state_id=ibkr-tickers-to-postgres-3 stdio=stderr
2022-08-13T20:55:01.294920Z [info     ]     from .application import Application cmd_type=extractor name=tap-ibkr-tickers run_id=67ec74b6-9c9d-4a22-a6bc-10f2655b0d72 state_id=ibkr-tickers-to-postgres-3 stdio=stderr
2022-08-13T20:55:01.295013Z [info     ]   File "/usr/local/lib/python3.10/dist-packages/poetry/console/application.py", line 5, in <module> cmd_type=extractor name=tap-ibkr-tickers run_id=67ec74b6-9c9d-4a22-a6bc-10f2655b0d72 state_id=ibkr-tickers-to-postgres-3 stdio=stderr
2022-08-13T20:55:01.295098Z [info     ]     from poetry.poetry import Poetry cmd_type=extractor name=tap-ibkr-tickers run_id=67ec74b6-9c9d-4a22-a6bc-10f2655b0d72 state_id=ibkr-tickers-to-postgres-3 stdio=stderr
2022-08-13T20:55:01.295185Z [info     ]   File "/usr/local/lib/python3.10/dist-packages/poetry/poetry.py", line 4, in <module> cmd_type=extractor name=tap-ibkr-tickers run_id=67ec74b6-9c9d-4a22-a6bc-10f2655b0d72 state_id=ibkr-tickers-to-postgres-3 stdio=stderr
2022-08-13T20:55:01.295269Z [info     ]     from .packages import Locker cmd_type=extractor name=tap-ibkr-tickers run_id=67ec74b6-9c9d-4a22-a6bc-10f2655b0d72 state_id=ibkr-tickers-to-postgres-3 stdio=stderr
2022-08-13T20:55:01.295355Z [info     ]   File "/usr/local/lib/python3.10/dist-packages/poetry/packages/__init__.py", line 2, in <module> cmd_type=extractor name=tap-ibkr-tickers run_id=67ec74b6-9c9d-4a22-a6bc-10f2655b0d72 state_id=ibkr-tickers-to-postgres-3 stdio=stderr
2022-08-13T20:55:01.295438Z [info     ]     from .locker import Locker cmd_type=extractor name=tap-ibkr-tickers run_id=67ec74b6-9c9d-4a22-a6bc-10f2655b0d72 state_id=ibkr-tickers-to-postgres-3 stdio=stderr
2022-08-13T20:55:01.295522Z [info     ]   File "/usr/local/lib/python3.10/dist-packages/poetry/packages/locker.py", line 9, in <module> cmd_type=extractor name=tap-ibkr-tickers run_id=67ec74b6-9c9d-4a22-a6bc-10f2655b0d72 state_id=ibkr-tickers-to-postgres-3 stdio=stderr
2022-08-13T20:55:01.295604Z [info     ]     from poetry.repositories import Repository cmd_type=extractor name=tap-ibkr-tickers run_id=67ec74b6-9c9d-4a22-a6bc-10f2655b0d72 state_id=ibkr-tickers-to-postgres-3 stdio=stderr
...
a
Looks like you are pretty close. Rather than running EL and T via 
meltano elt
can you try with the new 
meltano run
syntax: 
meltano run  tap-ibkr-tickers target-postgres dbt-snowflake:run
?
Or, you can run 
meltano elt
without the transform step (--transform=skip) and then follow with 
meltano run dbt-postgres:run
?
e
I think I found one of the problems, and fixed it.. basically when you use juju you're running hooks as root.. but for security I try to use everything as user 
ubuntu
and avoid running meltano as root (it's a container but... still 🙂 )
anyway.. I now can 
sudo apt install python3-poetry
in order to get poetry globally installed and it seemed to unlock further progress.. I will likewise take your suggestions today and try to see how far I get... but it's looking almost there ! the one exception is DBT.. every time I run the dbt transforms.. even with a brand brand new project .. it's calling out for the old 
dbt
, and I cannot fathom how or why.. except if it's something in the tap (but I looked and don't see any connection there)
I also gave up and started using 
pipx
for the meltano installation.. which seemed to improve the "running as non-root" issue
okay I found the issue thanks to the tip to 
meltano run
, thank you
Copy code
time=2022-08-14 20:15:38 name=tap-ibkr level=ERROR message=Exception occurred, stopping.. (psycopg2.errors.UndefinedTable) relation "analytics.tickers_query_time_stamp_last" does not exist cmd_type=elb consumer=False name=tap-ibkr-tickers producer=True stdio=stderr string_id=tap-ibkr-tickers
in past DBT was auto creating these tables, as I recall.. so I am unsure what or how to change it back to that behavior other than generate the schemas myself in a step somewhere
this problem stems from the fact that.. I wrote a tap but unsure how to tell it "tap only this end point"
that table isn't filled until ticker symbols are downloaded so.. of course the tables empty
okay I think I've refactored the tap now to just.. ignore the absence of the table successfully
once I have the tap tested as working.. I will add on the DBT part once again
thanks AJ
some feed back here is.. I think if you use the GUI it's just very very hard to decipher what's gone wrong.. and likely if more info is revealed with some CLI commands those should be surfaced or prompted for me as an end user to try
but some of this is likely my taps fault.. not being well written to handle certain states like.. if a stream is tried before it's ready
Open in Slack
PreviousNext
Powered by