https://linen.dev logo
Join Slack
Powered by
I was hoping to assemble a python project includin...
# troubleshooting
a
I was hoping to assemble a python project including meltano, dagster and dbt core following MDS approaches. However it looks like there is a dependency clash between meltano and dbt-core on 
jsonschema
(in dbt-core's case, via 
hologram
). Guess I'm out of luck until this is resolved on the dbt side? pip message in the thread 👇
Copy code
Because no versions of hologram match >0.0.14,<0.0.15
 and hologram (0.0.14) depends on jsonschema (>=3.0,<3.2), hologram (>=0.0.14,<0.0.15) requires jsonschema (>=3.0,<3.2).
And because hologram (0.0.15) depends on jsonschema (>=3.0,<4.0)
 and dbt-core (1.4.4) depends on hologram (>=0.0.14,<=0.0.15), dbt-core (1.4.4) requires jsonschema (>=3.0,<4.0).
And because meltano (2.16.1) depends on jsonschema (>=4.17,<5.0)
 and no versions of meltano match >2.16.1,<3.0.0, dbt-core (1.4.4) is incompatible with meltano (>=2.16.1,<3.0.0).
v
We recommend installing meltano in a seperate venv. Meltano then manages seperate venvs for all of the different extractors, loaders, and transformers (dbt) so that you don't hit any dependency issues
https://docs.meltano.com/concepts/python_virtual_environments is a good read on Meltano's approach here I think
a
That is cool, I saw the .venv subfolders but never figured each extractor has its own venv too. I am using poetry to manage the venv so far. So rather than adding dbt /dagster as a top level requirement in pyproject.toml 
poetry add dagster
etc, I should just add meltano at the top level (
poetry add meltano
) then add dagster and dbt as meltano utilities?
v
Yeah that works! We tend to recommend folks use 
pipx
, at the "top level" you'd have a global 
venv
that's just for 
meltano
, you could use poetry and just have 
meltano
as a dependency 🤷
a
Just trying some of this out now, ideally I would be able to define a DAG from a meltano run and onto some dbt transformations. I can see the meltano elements in dagster, but not sure if I can get the dbt models visible in dagster too using this method?
Open in Slack
PreviousNext
Powered by