Hey! I'm wondering what are the tactics to running meltano from a programmatic perspective (pure library). The docs mostly speak to running from the cli commands. Is there a clean wrapper to just call particular taps directly (and at scale to all the different taps, not just hooking into one). For now I call a subprocess, but that seems a little hacky...

yeah that's what we all do right now, we should just make a little python wrapper to make it nicer

Dang, appreciate the response. I have tried to load the packages dynamically, but all the tap python modules have a lot of different structures. No one tried that you know of?

Instead of subprocess, you can import the meltano cli click app and run it

I have tried to load the packages dynamically, but all the tap python modules have a lot of different structures.

They also have different dependencies, so that approach is essentially fraught.

I'm intrigued in this conversation. So what is the best practice if you want to have lots of different pipelines, you have a handful of taps and a handful of targets to be able to change the pipeline you're running? I have it currently set up where I've got a large Meltano project with all of them installed, and then I either swap out the meltano.yml file or am calling etl scripts at runtime. Is there a better philosophy to approach scaling up?

We handle this in our orchestration. We generate workflows that perform different EL tasks. Our

meltano.yml

file is large (because it has numerous taps and targets) although we're thinking of splitting it into multiple files and assembling it programmatically. I think Meltano only conceives of a very basic project so most complexity is better handled by outside processes.

Thanks Ellis for the response. How would you approach this portion of it:

splitting it into multiple files and assembling it programmatically

We haven't done this yet so it's an open question. Probably would start with something like

yq

or just use Python in our CI

We almost always use subprocesses and call Meltano But I honestly am liking

from meltano.cli.cli import cli; cli(["--version"])

It seems reasonable to commit to keeping

from meltano.cli.cli import cli; cli(["--version"])

stable 👍