I have a problem where my Snowflake account has no default role, I therefore need to set the role parameter. It took me a while to work out what was going on here and why I couldn't connect to Snowflake properly. So if I run the following command, I can see there is no role parameter, I realise it is optional but other optional parameters are coming through.

(.venv) [myuser@myhost myvenv]$ meltano config target-snowflake
{
 "account": "myaccount.ap-southeast-2",
 "dbname": "mydbname",
 "user": "my_user",
 "password": "my_password",
 "warehouse": "my_warehouse",
 "file_format": "MYSCHEMA.MY_CSV_FORMAT",
 "default_target_schema": "MYSCHEMA,
 "s3_bucket": "mysnowflakebucket-sc",
 "stage": "SALESFORCE_SANDBOX_V4.S3_RESTRICTED_STAGE",
 "batch_size_rows": 100000,
 "flush_all_streams": false,
 "parallelism": 0,
 "parallelism_max": 16,
 "disable_table_cache": false,
 "add_metadata_columns": false,
 "hard_delete": false,
 "data_flattening_max_level": 0,
 "primary_key_required": false,
 "validate_records": false,
 "no_compression": false
}

Note: There is no role parameter discovered. This is an important parameter as it is required to get my Warehouse, Database etc in Snowflake. To resolve I had to put the following logic into my meltano.yml. Specifically the config: role: DUMMY_ROLE

loaders:
 - name: target-snowflake
   variant: transferwise
   pip_url: git+<https://github.com/transferwise/pipelinewise-target-snowflake>
   config:
     role: DUMMY_ROLE
   settings: null
   disable_collection: true

Now if I re-run the config, my parameter is available and will grab the value from my .env which is good. The value DUMMY_ROLE is not used.

(.venv) [myuser@myhost myvenv]$ meltano config target-snowflake
meltano config target-snowflake
{
 "account": "myaccount.ap-southeast-2",
 "dbname": "mydbname",
 "user": "my_user",
 "password": "my_password",
 "warehouse": "my_warehouse",
 "file_format": "MYSCHEMA.MY_CSV_FORMAT",
 "default_target_schema": "MYSCHEMA,
 "s3_bucket": "mysnowflakebucket-sc",
 "stage": "MYSCHEMA.S3__STAGE",
 "batch_size_rows": 100000,
 "flush_all_streams": false,
 "parallelism": 0,
 "parallelism_max": 16,
 "disable_table_cache": false,
 "add_metadata_columns": false,
 "hard_delete": false,
 "data_flattening_max_level": 0,
 "primary_key_required": false,
 "validate_records": false,
 "no_compression": false,
 "role": "my_role"
}

Thoughts? Is this expected behaviour or does the role parameter need to be added somewhere? I do have a work-around as described, but this is not what I expected. Thanks Steve

Indeed - that looks like an omission on our end. I’ll make an issue 👍

Hi @taylor, I believe there is another missing parameter for

target-snowflake

in https://www.meltano.com/discovery.yml . You mentioned you were going to make an issue for the missing

role

environment variable / parameter, could we also please add

batch_wait_limit_seconds

too.

batch_wait_limit_seconds

Integer(Default: None)

Maximum time to wait for batch to reach batch_size_rows.

Thanks Steve

thanks for the ping! I’ve updated the issue here https://gitlab.com/meltano/meltano/-/issues/3015