Hi All, It appears that the recent update to target-snowflake

0.17.2

has updated the Meltano SDK to version

0.46

which included change https://github.com/meltano/sdk/pull/3020 to pull schema name from the stream name when the

default_target_schema

config is set. Previously I was using the

default_target_schema

setup to strip the schema name from the source stream and force target to a specific schema in my snowflake database. From what I can see there is no way to force a schema at the target level with this change when your stream name has a hyphen in it as the target takes the schema name from the SDK method prior to checking the

schema

config setting. https://github.com/MeltanoLabs/target-snowflake/blob/main/target_snowflake/sinks.py#L56 Does anyone have any suggestions around the best way to handle this, I did take a look at stream maps on the target (source is not SDK based) and setting the map on the base target didn't seem to propagate to all my children target of which I have a few?

I'm actually thinking of reverting this change (cc @Reuben (Matatika) in case you have any alternative ideas)

My only thought was that the targets schema config should possibly override the one found in the stream name when set but this would also be a breaking change on existing users

Typical use cases for data migration are:

default_schema.source_table

->

default_schema.target_table

source_schema.source_table

->

default_schema.target_table

or

target_schema.target_table

I can only think one scenario where

source_schema.source_table

->

default_schema.source_schema_target_table

will be acceptable behaviour. I.e. when I have existing

target_table

which I don't want to overwrite and I am unable to create a staging area or I am forced to use the

default_schema

To me, presence of

default-target-schema

should override the incoming db schema name. Unfortunately singer has no way to transmit the db schema name, so the convention of using the

source_schema.source_table

is followed. As I am new to meltano, I will expect that if

default-target-schema

exists, then the target will use the table name as

default_target_schema.target_table

and not

default_target_schema.source_schema_target_table

Unfortunately I am not sure how many people depend upon this behaviour, so a breaking change will be risky.

In our case, some incoming stream names had hyphens in that we did not want to be loaded as

<schema>-<table>

. Now, setting

default_target_schema

means that the stream names are interpreted at face-value, rather than trying to parse out a target schema/table name for each. > the targets schema config should possibly override the one found in the stream name when set but this would also be a breaking change on existing users I think that was the intention of the change: • stream name:

foo-bar

, `default_target_schema`: not set ◦ schema name:

foo

◦ table name:

bar

◦ identifier:

foo.bar

• stream name:

foo-bar

, `default_target_schema`:

foo

◦ schema name:

foo

◦ table name:

foo-bar

◦ identifier:

foo.foo-bar

• stream name:

foo-bar

, `default_target_schema`:

baz

◦ schema name:

baz

◦ table name:

foo-bar

◦ identifier:

baz.foo-bar

Yes, it is breaking for some use-cases. > From what I can see there is no way to force a schema at the target level with this change when your stream name has a hyphen in it as the target takes the schema name from the SDK method prior to checking the

schema

config setting If you want to "force" the schema, you should just have to set

default_target_schema

. Is this not what you are seeing? With

default_target_schema

In prior versions, the target would resolve the schema name in the same way, but the table name was still taken as the last hyphen-separated "part" of the stream name (rather than the raw stream name now).

My needs and assumptions align exactly with @Satish Bhat. also as I use Snowflake as a target having a hyphen in a table name means having to quote qualify every usage of that table name which makes it much messier to work with. The Singer spec seems to have had a long running convention of schema-table naming and this seems to go against that. In my case the

target-snowflake

has a config for

schema

separate to

default_target_schema

however the target functionality appears to preference the SDK over the config which if reversed would allow an override. https://github.com/MeltanoLabs/target-snowflake/blob/72ffd552a1a4f3e08f8691efb482f1301fa41312/target_snowflake/sinks.py#L57

I just published v0.46.4 reverting the table name change when

default_target_schema

is set. Thank you for your patience and sorry about the disruption 🙏.

I'll take the blame for the initial suggestion! 😬😅

@Edgar Ramírez (Arch.dev) any idea when this will flow through to target-snowflake?

@Chad This is already available if you reinstall the plugin

Thanks