Hey guys. I was wondering if there is any way to save the State (State backend) in Snowflake and not in AWS, Azure and co. Has anyone tried that already?

It's not supported at the moment, but I think it'd not be a great choice for storing state since the requirement is basically a key-value store. For currently supported stores, the key is either a file path, an object key or an indexed column value. Since Snowflake doesn't really have indexes, a full-table scan would be performed at least a couple times for every pipeline run. You could still try it our at your own peril, though 😅. 1. Install

snowflake-sqlalchemy

in the same venv as meltano 2.

meltano config meltano set database_uri snowflake://{user}:{password}@{account}

3.

meltano config list

. This would run db migrations on snowflake, this is the point where things are most likely to break imo 4. Use the default state store, i.e. the system db which now points to Snowflake

great thankyou! alright, yeah i see the concern 😄

FYI I once tried the above steps, and things did indeed break during migrations, as @edgar_ramirez_mondragon predicted. The root cause was lack of support by

snowflake-sqlalchemy

for certain

sqlalchemy

functions that

alembic

uses (even after I disabled the indexing-related migrations).