https://linen.dev logo
#plugins-general
Title
# plugins-general
j

joshua_janicas

09/11/2023, 2:40 PM
Hello! Not sure if this is the right place to ask: 1) I have been using with success Meltano's variant of
target-snowflake
and trying to have discussions with my coworkers as how a production environment would look like. Some members from our "SecurityOps" team are asking me questions about security (how secure?) the data being transferred which I don't have answers for at the moment. As part of our analytical warehouse we have some personal identifying information (such as as customer first / last names , phone numbers) that SecOps is extra concerned about. The biggest Q is how does a the target send the compressed JSON files to Snowflake - is it over a secure connection? The typical connection looks like
https://<orgname>-<account_name>.<http://snowflakecomputing.com|snowflakecomputing.com>
so I assume the HTTPS connection alone should satisfy SecOps? I've been trying to look for documentation on Meltano/target-snowflake:meltano GitHub's page as well as Snowflake for more details around these connections but I was hoping for someone could point me in the right direction. Some additional notes: • We're mostly a Microsoft shop, so we are using Azure as our cloud computing service • Due to data residency requirements, we are currently forced to use Meltano's core solution via docker and self hosting. • We plan to have one Snowflake "account" per region (one for US/one for Canada etc.) 2) As an extension of Q1), I know that Azure has a Private Link functionality which Snowflake supports (Requiring the right Snowflake tier of course). Has anyone ever used private link functionality before with Meltano? If SecOps forces me to use private link functionality, the typical connection would start to look like
https://<orgname>-<account_name>.<http://privatelink.snowflakecomputing.com|privatelink.snowflakecomputing.com>
(https://docs.snowflake.com/en/user-guide/organizations-connect#private-connectivity-urls). Following this logic, in order to have Meltano connect to it, the env variable
TARGET_SNOWFLAKE_ACCOUNT='abc-xyz'
I would think it would need to be changed to
TARGET_SNOWFLAKE_ACCOUNT='abc-xyz.privatelink'
to compensate? This is assuming that all the infrastructure set up on Snowflake/Azure is properly set up.