For anyone here using Prefect instead of Airflow how did you Meltano #announcements

For anyone here using Prefect instead of Airflow: ...

julian_knight

07/17/2020, 5:33 PM

For anyone here using Prefect instead of Airflow: how did you get around the version conflict with

urllib3

between Meltano and Prefect?

adam_roderick

07/17/2020, 7:58 PM

We use prefect and meltano. We also use dbt, which has yet another dependency conflict! The solution was pretty simple for us:

adam_roderick

07/17/2020, 7:59 PM

We have a main virtual environment where we install prefect and other orchestration-related components. Meltano and dbt are relegated to their own venvs. So our structure looks like this:

Copy code

venv/
  meltano/
  dbt/

adam_roderick

07/17/2020, 8:00 PM

During development time, when working with meltano, we activate the meltano env so that we can run commands with

meltano

adam_roderick

07/17/2020, 8:00 PM

At other times, we run commands using

venv/meltano/bin/meltano

, which will run in the context of the meltano venv, even though we have the main venv activated

julian_knight

07/17/2020, 8:15 PM

Nice, thanks. So you run multiple venvs in production too? Do you have conflicts, e.g. triggering

meltano elt

shell tasks from prefect's venv?

adam_roderick

07/17/2020, 8:25 PM

We do run multiple venvs in production

adam_roderick

07/17/2020, 8:26 PM

We have no conflicts. Because meltano is not installed in prefects venv, we executed it with a command like

venv/meltano/bin/meltano

. Because we execute it that way, that command runs in the context if meltano's venv, even though meltano's venv is not activated.

julian_knight

07/17/2020, 8:27 PM

that makes sense. Thanks for your help!

douwe_maan

07/17/2020, 8:34 PM

Note that activating a venv literally just puts

<venv>/bin

at the front of the

PATH

, so executing an executable in

<venv>/bin

directly is totally valid and equivalent 🙂

7 Views

Open in Slack

Previous Next