Hey Community after extensively exploring Meltano, we are convinced that it can serve as an excellent dataops tool for our organisation. However, we are seeking practical, hands-on resources, tutorials, or guides that demonstrate how to effectively utilize Meltano in a production environment. While we have reviewed the documentation, we would greatly appreciate a more practical approach that showcases real-world use cases. Currently, we employ Airflow to orchestrate our jobs and are considering using the KubernetesPodOperator to incorporate Meltano. Our plan involves treating each Meltano call as a separate tap or target action. For instance, we would use the operator to execute a tap task, extract data, and write it to an intermediate storage such as Amazon S3. We are curious if anyone has already implemented Meltano in a similar manner and could share their experiences or guide in the right direction . Specifically, we are interested in understanding how others have managed state configurations, updated metadata, and ensured synchronization. Any insights or advice from your own implementation would be invaluable to us.

This blog post might be relevant https://meltano.com/blog/deploying-meltano-for-meltano/. Although, once Meltano Cloud was released we moved all of our internal pipelines there to avoid extra infra. It was running nicely for over a year though.

Your idea to run a tap + target pair e.g. tap-xyz target-s3-csv seems like the right approach. Is S3 your final destination or do you mean you want it in S3 as an intermediate?

Hey @pat_nadolny thanks for response will go through the link . The ideal is similar to what you have guessed , so we plan to have a base meltano image which has all the global configs and the intermediate target (i.e. S3 csv or json) set as default. Then all other taps will inherit this base image and extract the data to the s3 as intermediate thus accomplishing the EL of the ELT. The transform part we will look to employ DBT and then send it to the target .