Not sure which is the best place to ask this. (It is relevant to Meltano) Currently, we’re using Snowpipes to fetch data from S3 into snowflake and then transform data via Snowflake tasks. For other data EL from External DBs, we have Meltano Pipelines setup from MySQL to snowflake. My question is, Can I replace Snowpipes with Meltano to fetch E from S3 and L into snowflake? Can meltano work reliably when pulling data from S3 into Snowflake ? currently our S3 event based Snowpipe triggers work very well. Secondly, if I remove the transformation tasks from Snowflake and want to use Meltano only to do Transformations, will it work. Can I run meltano in Transform-Only mode

Can I replace Snowpipes with Meltano to fetch E from S3 and L into snowflake? Can meltano work reliably when pulling data from S3 into Snowflake ? currently our S3 event based Snowpipe triggers work very well.

Yes, I have no idea how good Snowpipes is or isn't but yes Meltano can do this

Secondly, if I remove the transformation tasks from Snowflake and want to use Meltano only to do Transformations, will it work. Can I run meltano in Transform-Only mode

Yes,

meltano invoke dbt:run

or whatever transformer you're using

Thanks @visch for the leads, for the S3 replacement, I am assuming you are talking about https://hub.meltano.com/taps/s3-csv , but our data is in JSON format

Is there a tap written for s3 that pull json data? hmm not sure

I searched, there’s no “official” s3 json tap