Is anyone running Meltano in kubernetes via Argo W...
# infra-deploymentm
Matt Menzenski
12/02/2022, 2:34 AMIs anyone running Meltano in kubernetes via Argo Workflows (as opposed to Airflow)?
e
edgar_ramirez_mondragon
12/02/2022, 3:41 PMI think @stephen_bailey was using it at some point
edgar_ramirez_mondragon
12/02/2022, 3:41 PMm
Matt Menzenski
12/02/2022, 3:42 PMthank you!!
Matt Menzenski
12/22/2022, 6:00 PMRelated to running Meltano in Argo Workflows - I see that for Dagster there is a utility plugin https://hub.meltano.com/utilities/dagster while for Airflow there is both a utility plugin https://hub.meltano.com/utilities/airflow and an orchestrator plugin https://hub.meltano.com/orchestrators/airflow and a file plugin https://hub.meltano.com/files/files-airflow .
If I’m interested to add a plugin for Argo Workflows, is there an obvious starting point? I don’t see clear support for plugin types “file”, “orchestrator”, or “utility” in the SDK docs https://sdk.meltano.com/en/latest/ .
d
douwe_maan
12/22/2022, 6:04 PMFor plugins other than Singer taps, targets, and mappers, you’ll want to use the Extension Development Kit: https://edk.meltano.com/en/latest/. https://github.com/meltano/airflow-ext/ and https://github.com/quantile-development/dagster-ext are both good examples that also wrap the necessary project files. The Airflow orchestrator plugin is pre-EDK so you can ignore that one.
m
Matt Menzenski
12/22/2022, 6:06 PMaha, so I was looking at the SDK when I should have been looking at the EDK. Thanks!
d
douwe_maan
12/22/2022, 6:13 PM@Matt Menzenski I’m updating the SDK docs to make the distinction clear: https://github.com/meltano/sdk/pull/1293/files