Hello, I'm struggling with understanding how to incorporate the meltano dag into an existing Airflow installation. it feels as though meltano doesn't give much thought towards integrating with airflow (or maybe other orchestrators) in a managed or k8s environment. I'm trying to accomplish this without having to write a custom dag since meltano can generate an airflow one, but that seems to be with the condition that your airflow and meltano binaries are on the same host. Is there something maybe i'm missing about the generator or is my hunch correct that this isn't really intended to be separate components? For context, we are running this:

meltano add files files-airflow

Then when I try to add the dag to Airflow it complains about DAG import error with:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/local/lib/python3.11/subprocess.py", line 1955, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'meltano'

Hi @Buddy Ruddy 👋

it feels as though meltano doesn't give much thought towards integrating with airflow (or maybe other orchestrators) in a managed or k8s environment.

Agreed, though I don't know how the opposite would look like? i.e. what is Meltano (or its docs) missing that would make integration smoother? To me it feels that there's just so many ways of deploying and operating an orchestrator that it's impossible to write a sufficiently general guide, much less the utilities to translate Meltano's schedules into the orchestrator in question. That said, I think we're happy to consider any type of improvement.

I would think providing meltano as more of a library rather than a cli which creates a dependency

and i mean that as in addition to the cli, so that when its favorable to import meltano into something else, it's easier to do so

if this is already available, id be happy to try that method, but unfortunately the documentation doesn't illustrate that from what i can tell

It is not available at the moment, and though you could do it in theory today, it's not documented and it's hard to tell what problems you might run into.