I have a work project I’m trying to decide is worth it or not: 1. build a custom meltano tap for https://simpleanalytics.com (they seem to have a friendly API that can spit out csv/parquet/json) 2. Use this to ETL our data into our BQ, on schedule Re 1- I haven’t built an ETL pipeline before. What’s the lift like here? In general I feel like maintaining pipelines is not the one. Re 2- what are the best options for scheduling right now? I see meltano cloud is in Alpha? What else is easy to spin up?

I wrote the following relevant article: https://medium.com/gooddata-developers/extending-ci-cd-data-pipeline-with-meltano-7de3bce74f57 It is still a naive demo but shows the way. The scheduling is done in Gitlab but should be done e.g. by Dagster or Airflow. I plan to extend the pipeline accordingly in near future. The ultimate solution is to deploy all components to Kubernetes or as serverless functions. I would like to end up the demo like this. It still will be orchestrated by Gitlab, but instead of running Meltano/dbt/... in Gitlab workers, Gitlab will deploy artifacts to k8s/serverless. Anyway, the Gitlab CICD way is IMO correct and it is very easy to set up.

https://sdk.meltano.com/en/latest/ for the custom tap @archie_sarre_wood