Not sure where to put this, so I'll put it here. I would appreciate any suggestions for orchestrating or scheduling Meltano jobs at many locations that do not have reliable network connectivity to a central site, but which can also be configured from the central site. Think of a centralized management plane to push the configurations, but local control planes to execute the jobs if connectivity to the main site is down. It doesn't look like Airflow supports this type of distributed scheduling / orchestration.

Hey @fred_reimer, we have been thinking a bit about this too lately 👏 As far as I can see, none of the orchestrators readily support routing and running individual tasks in specific locations, and this was definitely a big pain point for me in previous roles. Super excited to hear that you are seeing this issue in your work too 😅 However it is a weighty problem to solve, so it would really help to have your use case and feedback added to the issue to help us figure out if its something Meltano could help with 🙂🙏

Can you send me a DM when you have a chance to chat? We can summarize back here for the generic use-case if desired.

very simple version of this is to have your "deployment" be setting up a cron on your down stream sites

Very interesting topic! Copying @jo_pearson, @connor_flynn, and @kai_yokoyama

@aaronsteers FYI 🙂