joshua_janicas
09/11/2023, 2:40 PMtarget-snowflake
and trying to have discussions with my coworkers as how a production environment would look like. Some members from our "SecurityOps" team are asking me questions about security (how secure?) the data being transferred which I don't have answers for at the moment. As part of our analytical warehouse we have some personal identifying information (such as as customer first / last names , phone numbers) that SecOps is extra concerned about. The biggest Q is how does a the target send the compressed JSON files to Snowflake - is it over a secure connection? The typical connection looks like https://<orgname>-<account_name>.<http://snowflakecomputing.com|snowflakecomputing.com>
so I assume the HTTPS connection alone should satisfy SecOps? I've been trying to look for documentation on Meltano/target-snowflake:meltano GitHub's page as well as Snowflake for more details around these connections but I was hoping for someone could point me in the right direction.
Some additional notes:
• We're mostly a Microsoft shop, so we are using Azure as our cloud computing service
• Due to data residency requirements, we are currently forced to use Meltano's core solution via docker and self hosting.
• We plan to have one Snowflake "account" per region (one for US/one for Canada etc.)
2) As an extension of Q1), I know that Azure has a Private Link functionality which Snowflake supports (Requiring the right Snowflake tier of course). Has anyone ever used private link functionality before with Meltano? If SecOps forces me to use private link functionality, the typical connection would start to look like https://<orgname>-<account_name>.<http://privatelink.snowflakecomputing.com|privatelink.snowflakecomputing.com>
(https://docs.snowflake.com/en/user-guide/organizations-connect#private-connectivity-urls). Following this logic, in order to have Meltano connect to it, the env variable TARGET_SNOWFLAKE_ACCOUNT='abc-xyz'
I would think it would need to be changed to TARGET_SNOWFLAKE_ACCOUNT='abc-xyz.privatelink'
to compensate? This is assuming that all the infrastructure set up on Snowflake/Azure is properly set up.Peter
03/12/2024, 10:58 PMjoshua_janicas
03/12/2024, 10:59 PMPeter
03/13/2024, 1:53 AMjoshua_janicas
03/13/2024, 12:27 PM