Hi, I am just starting to explore Meltano We need to perform daily incremental sync from MySQL into Snowflake. We started off with Airbyte on Kubernetes, but it currently we’re stuck due to some issues with MySQL connection where processing just pauses after fetching 150k records With no way forward, looking into Meltano if it solves problems. Desired Deployment Method : Kubernetes Deployment Runtime: Docker Containers Source: Mysql Target: Snowflake Mode: Incremental loads Does anybody have experience with such setup, atleast with MySQL as source ?

Hi @binoy_shah! All of that is possible with Meltano. For the Kubernetes + Docker deployment you can probably reference the great writeup by my colleague @ken_payne https://meltano.com/blog/deploying-meltano-for-meltano/ as well as the source code https://gitlab.com/meltano/squared. The default MySQL extractor and Snowflake loader variants are robust in my experience simple smile

Thank you @edgar_ramirez_mondragon, I started looking into the @ken_payne article, but we have a very stable Kube Pipeline with Helm Charting already achieved.. So all I need to do is correctly inject Secure ENV variables for connectivity, but I am still new to Singer based Extraction itself so trying to find good sources that teach me basics..

Hi @binoy_shah we run meltano on kubernetes, so this part will definitely work. You may still run into specific issues with the tap or target of course. To run meltano in a pod we set the required extractor / target environment variables before running Meltano (from secrets), but you can also use Meltano yml files to configure the environment. We recently created a getting started guide that explains how to do this (it's in the context of our platform, but it exactly what you need to do, so might help you) https://www.matatika.com/docs/getting-started/running-your-data-import-locally and a video if you prefer to watch something

Thank You @aaron_phethean