https://linen.dev logo
Join Slack
Powered by
I'm getting a version conflict. metlano requires s...
# announcements
a
I'm getting a version conflict. metlano requires snowflake-connector-python 1.6.10, while dbt (installed separately) requires a later version. Any advice, short of separate virtual environments?
I'm working on packaging things for a deployment scenario, and I think I'm limited to a single python env
d
@adam_roderick Hmm. Meltano installs all of its plugins in their own virtual environments inside 
.meltano
, so there should be no conflicts. Are you on Meltano v1.32.1 (https://meltano.com/blog/2020/05/15/now-available-meltano-v1-32-1/)? You may be affected by the bug fixed in there
If that's not it, can you share the error message/stack trace so that I can see where things are installed and running from?
a
I am on that version, but installed dbt into the virtual env separate--not as a meltano plugin
Command: 
meltano elt <tap> <target>
results in:
```Traceback (most recent call last): File "/mnt/c/dev/datateer/dtprod-pipeline/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 584, in _build_master ws.require(requires) File "/mnt/c/dev/datateer/dtprod-pipeline/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 901, in require needed = self.resolve(parse_requirements(requirements)) File "/mnt/c/dev/datateer/dtprod-pipeline/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 792, in resolve raise VersionConflict(dist, req).with_context(dependent_req) pkg_resources.ContextualVersionConflict: (snowflake-connector-python 2.2.1 (/mnt/c/dev/datateer/dtprod-pipeline/venv/lib/python3.6/site-packages), Requirement.parse('snowflake-connector-python==1.6.10'), {'meltano'}) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/mnt/c/dev/datateer/dtprod-pipeline/venv/bin/meltano", line 6, in <module> from pkg_resources import load_entry_point File "/mnt/c/dev/datateer/dtprod-pipeline/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3252, in <module> @_call_aside File "/mnt/c/dev/datateer/dtprod-pipeline/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3236, in _call_aside f(*args, **kwargs) File "/mnt/c/dev/datateer/dtprod-pipeline/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3265, in _initialize_master_working_set working_set = WorkingSet._build_master() File "/mnt/c/dev/datateer/dtprod-pipeline/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 586, in _build_master return cls._build_from_requirements(requires) File "/mnt/c/dev/datateer/dtprod-pipeline/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 599, in _build_from_requirements dists = ws.resolve(reqs, Environment()) File "/mnt/c/dev/datateer/dtprod-pipeline/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 787, in resolve raise DistributionNotFound(req, requirers) pkg_resources.DistributionNotFound: The 'snowflake-connector-python==1.6.10' distribution was not found and is required by meltano (venv) adam@adampc:/mnt/c/dev/datateer/dtprod-pipeline/meltano$ meltano elt tap-indeed target-postgres Traceback (most recent call last): File "/mnt/c/dev/datateer/dtprod-pipeline/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 584, in _build_master ws.require(requires) File "/mnt/c/dev/datateer/dtprod-pipeline/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 901, in require needed = self.resolve(parse_requirements(requirements)) File "/mnt/c/dev/datateer/dtprod-pipeline/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 792, in resolve raise VersionConflict(dist, req).with_context(dependent_req) pkg_resources.ContextualVersionConflict: (snowflake-connector-python 2.2.1 (/mnt/c/dev/datateer/dtprod-pipeline/venv/lib/python3.6/site-packages), Requirement.parse('snowflake-connector-python==1.6.10'), {'meltano'}) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/mnt/c/dev/datateer/dtprod-pipeline/venv/bin/meltano", line 6, in <module> from pkg_resources import load_entry_point File "/mnt/c/dev/datateer/dtprod-pipeline/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3252, in <module> @_call_aside File "/mnt/c/dev/datateer/dtprod-pipeline/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3236, in _call_aside f(*args, **kwargs) File "/mnt/c/dev/datateer/dtprod-pipeline/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3265, in _initialize_master_working_set working_set = WorkingSet._build_master() File "/mnt/c/dev/datateer/dtprod-pipeline/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 586, in _build_master return cls._build_f…
I missed the last line: 
pkg_resources.DistributionNotFound: The 'snowflake-connector-python==1.6.10' distribution was not found and is required by meltano
Or, is there a way to exclude the snowflake plugin?
d
I think the package is unnecessary if you're not using Meltano's Analyze UI with Snowflake, which I don't think anyone is doing today, and which isn't our focus for the moment, so I wouldn't mind removing it if that helps you. But I think you're asking for trouble by installing multiple packages in the same venv, because I can't promise similar conflicts won't arrive in the future
Would it be an option to install dbt as a Meltano plugin? You can interact with it as usual as long as you run 
meltano invoke dbt ...
instead of plain 
dbt ...
a
I think so, if I can control the dbt version. Can I?
d
Ah yeah, for sure! Run 
meltano add transformer dbt
, then edit the 
pip_url
in 
meltano.yml
to point at a different version, and run 
meltano install
🙂
a
Thanks! I'll give that a try
d
You're not bound to the specific plugins versions specified in 
discovery.yml
, that just lists what packages and versions are known to work, but if a later version is known to work, we should update it there 🙂
So if you get it working with a newer 
dbt
, we can try updating the version there too!
a
discovery.yml
contains all the defaults?
d
Yep
I updated this section of the docs on Friday: https://meltano.com/docs/contributor-guide.html#known-plugins
a
ah great
d
I'll add a section to the homepage/docs on the concept of the Meltano project and how 
meltano.yml
works and can be used in https://gitlab.com/meltano/meltano/-/issues/2035
a
dbt's latest version (0.16.1) works just fine! I updated the pip_url, ran 
meltano invoke dbt --version
with success
d
Awesome! Are you trying to run it as part of 
meltano elt
as well, or only on its own?
a
On its own
Use case is several sources need to be refreshed before running the dbt transforms
d
Makes sense
Are you planning to run this on Airflow? The default DAG (and 
meltano schedule
) may not be too useful to you then, but you can just add your own in 
orchestrate/dags
, and run 
meltano elt
, 
meltano invoke dbt
and 
meltano invoke airflow
as you please
a
No, exploring a different direction than airflow
d
Cool, curious to hear where you end up 🙂
a
Yeah, I'll report back
Currently we are on Prefect. Also exploring AWS Step Functions
d
Prefect looks interesting; a Meltano Task would be pretty cool 🙂
a
@adam_roderick out of interest, how have you found Prefect? How do you think it compares to Airflow?
6 Views
Open in Slack
PreviousNext
Powered by