After a few weeks I executed (again) a pipeline locally and it fails:

(.venv) jacek@holly:~/work/src/panther-internal-analytics$ make extract_load 
cd "data_pipeline" && meltano --environment $ELT_ENVIRONMENT run tap-salesforce $MELTANO_TARGET
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Emitter.__init__() got an unexpected keyword argument 'request_timeout'
make: *** [Makefile:25: extract_load] Error 1

I deleted virtualenv and .meltano and installed everything from scratch, it did not help. Anyone hit this issue?

@edgar_ramirez_mondragon I believe we saw this recently with

tap-jaffle-shop-template

pip freeze result

You're using

snowplow-tracker==0.10.0

, which is surprising since the

Emitter

class for that version has the keyword argument `request_timeout`: https://github.com/snowplow/snowplow-python-tracker/blob/0.10.0/snowplow_tracker/emitters.py#L61

Meltano 2.15.3 To simplify onboarding, devs install meltano, dbt and some other libs into the same virtual environment

Perhaps we could get some useful information by running with

--log-level=debug

. Another thing to check is what the output of the following is:

python -c 'import inspect, snowplow_tracker; print(snowplow_tracker.__version__); print(inspect.signature(snowplow_tracker.emitters.Emitter))'

devs install meltano, dbt and some other libs into the same virtual environment

If you change the installation order that might fix it, i.e. installing meltano after dbt and other libs. I suspect dbt’s snowplow tracker requirement is the offender.

I figured it might be something like that, but the

pip freeze

results suggest that a valid version of

snowplow-tracker

is installed, so now I'm not sure

There is the result of the execution with --log-level-debug

We recommend installing Meltano into its own virtual environment to avoid dependency conflicts like this. Would pipx be an option for you @jan_soubusta? Anything installed with it will automatically get its own virtual environment.

If I understand the DOC well (and also an explanation from ChatGPT), I would still need to switch between such venvs using deactivate/activate commands, which would be sort of annoying....

That is not correct. When using

pipx

the packages installed become globally available. If I run

pipx install meltano

, then I can run

meltano

from within any venv (or outside of one) so long as there is not

PATH

conflict (e.g. from also installing

meltano

within a venv).

Let's say I need to install meltano and dbt and iterate quickly with commands

meltano run ...

and

dbt run

. If I run

pipx install meltano

and

pipx install dbt

, both (including conflicting dependencies) will be installed into the same global scope, right? So the problem persists, right?

@jan_soubusta No, by installing with pipx Meltano and dbt will each be installed in its own virtualenv (so no dependency conflicts) and their corresponding executable added to PATH, so you'll be able to call

meltano run

and

dbt run

outside any virtualenv (so no activation required)

Wow, that sounds really great! What a fool I have been! 😉

@edgar_ramirez_mondragon One question I’ve received from colleagues is that pipx can lead to conflicts in python versioning. Say, for example, pipx installed meltano is a pyenv with python 3.9 but I’m developing a meltano pipeline with python 3.10. What’s the best way to handle the potential pyenv conflicts?

@niall_keleher You can specify the Python version you want the venv to be created with the

--python

option. I actually use both pyenv and pipx and have a few Meltano versions installed side-by-side:

$ pipx list | grep meltano@cp
   package meltano 2.19.0 (meltano@cp310), installed using Python 3.10.8
    - meltano@cp310
   package meltano 2.19.0 (meltano@cp37), installed using Python 3.7.16
    - meltano@cp37
   package meltano 2.19.0 (meltano@cp38), installed using Python 3.8.14
    - meltano@cp38
   package meltano 2.19.0 (meltano@cp39), installed using Python 3.9.16
    - meltano@cp39

perfect! thanks for the tip @edgar_ramirez_mondragon!

Getting back to this issue.

pipx

does not provide the option

-r requirements.txt

. I cannot install each package separately. I no longer can use the workaround with two venvs - I want to use Dagster orchestrating both dbt and meltano from the same (v)env. Thinking about how to solve it...

@jan_soubusta Can you log an issue in the Meltano issue tracker to avoid initializing the tracker when usage stats are disabled? That may help fix this issue for folks that have no option but to share a venv with dbt.

https://github.com/meltano/meltano/issues/8256 Is there a chance that it will be delivered any time soon?

@jan_soubusta You might wanna take a look at https://github.com/meltano/meltano/issues/8256#issuecomment-1802552712

Cool! It's 10:30pm here in Prague. Tomorrow we have company offsite. If anyone else does not prepare a PR till Friday, I try my best and create one.

I think I'm running into the same issue, Meltano started failing for me 5 days ago with this error:

ImportError: cannot import name 'SelfDescribing' from 'snowplow_tracker' (/Users/jacob/.pyenv/versions/3.10.11/envs/peach-meltano/lib/python3.10/site-packages/snowplow_tracker/__init__.py)

Reverting from

meltano==2.20.0

to

meltano==2.19

in my requirements.txt file "fixed" the issue

The issue is accepting PRs. A community member volunteered but I have not heard back so removed their assignment, if anyone's interested in picking it up 🙂

Eh, sad. I try to find a time for this and develop a solution.

thanks @jan_deelstra!

@jan_soubusta Thanks for the interest in contributing! It's probably the

eslint

hook. FWIW I got npm 9.8.1, so that's probably it: https://github.com/npm/cli/issues/6665#issuecomment-1645728379

How to execute tests locally?