Hello all! I'll try to make this concise, but apologies if it is a bit long. I've used the Singer API to create a custom tap and integrate with Airbyte to run scheduled jobs to extract from a custom API and target a database. It was suggested that I check out Meltano, as the docs for Singer API are a bit off, I've had some minor issues with getting it to work, and Meltano was said to be much better. So, realizing the answers may be a bit biased, some questions: 1. Is Meltano worth the switch, realizing I just got started anyway? 2. Does it integrate with Airbyte well, or should I switch to Apache Airflow? 3. Virtual Environment a. I use pyenv, as it is used for some other projects b. Do I have to learn and switch to pipx and poetry? c. Can they coexist, or do I have to figure out some way to switch between them? d. Does Melanto require pipx and poetry? They seem pretty embedded. 4. ContainerizationI browsed through the docs, is each tap and target a separate container, or does Meltano bundle everything together into one container? Thanks for your insight!

Hey Fred! I'm not an expert in either, but in relatively similar situation as you. 1. I tried airbyte and discarded in favor of Meltano. 2. My understanding is that they aren't very compatible as Airbyte is moving away from Singer and Meltano is embracing/extending singer. 3. You don't have to manage the virtualenvs, meltano does it for you 4. I haven't tried the containerized approach, I'm running everything in virtualenv's and that part of everything seems to Just Work (tm)

On point 2, Meltano ships with Airflow out of the box https://meltano.com/docs/orchestration.html

Do I have to learn and switch to pipx and poetry?

Nope - Poetry is recommended and "works out of box" for our SDK, but you can also build your own

setup.py

install method if you prefer. Poetry and pipx help a bunch for people with less experience in

pyenv

and virtual environment management techniques. We do still recommend learning them, but it's not strictly required. Also - just as an fyi - when meltano installs something via

meltano add

or

meltano install

, it also creates and manages the virtual environments, to prevent each plugin from conflicting with the others.