Working with the built-in

target-snowflake

and noticed an error when not defining

SF_SCHEMA

in my

.env

file. If undefined, I get this critical error:

target-snowflake | ERROR Exception in schema_apply() while processing:
target-snowflake | {"type": "SCHEMA", "stream": "issue_labels", "schema": {"type": ["null", "object"], "properties": {"_sdc_repository": {"type": ["null", "string"]}, "id": {"type": ["null", "number"]}, "node_id": {"type": ["null", "string"]}, "url": {"type": ["null", "string"]}, "name": {"type": ["null", "string"]}, "description": {"type": ["null", "string"]}, "color": {"type": ["null", "string"]}, "default": {"type": ["null", "boolean"]}}}, "key_properties": ["id"]}
target-snowflake | 
target-snowflake | CRITICAL (snowflake.connector.errors.ProgrammingError) 002003 (02000): SQL compilation error:
target-snowflake | CRITICAL Schema 'COOPER_DB."tap_github"' does not exist or not authorized.
target-snowflake | CRITICAL [SQL: GRANT USAGE ON SCHEMA "COOPER_DB"."tap_github" TO ROLE SYSADMIN;]
target-snowflake | CRITICAL (Background on this error at: <http://sqlalche.me/e/13/f405>)

When I look at my database, it does create a schama called

TAP_GITHUB

, but it still fails. When I define

SF_SCHEMA

as

TAP_GITHUB

the error does not occur.

@michael_cooper Thanks for raising this! @nevin_morgan had found the same thing in https://meltano.slack.com/archives/CFG3C3C66/p1596728929419200?thread_ts=1596724128.413500&cid=CFG3C3C66, created an issue in their fork: https://gitlab.com/vividfront/meltano/target-snowflake/-/issues/1, and started looking into a solution, but I think it slipped off of both our radars. @nevin_morgan Do you have an update on this? Would it be easy to address this in

target-snowflake

itself, or would it make more sense to implement a new

value_processor: upcase_string

and add it to

target-snowflake

schema

?

It isn't completely off my radar I just am stuck at the moment trying to get my GA tap running. I hacked it to uppercase the internal representation which works for loading the data, but then DBT fails to manipulate it

@nevin_morgan Yeah, it would, because when

dbt

reads the loader's

schema

to set its own

source_schema

(https://gitlab.com/meltano/meltano/blob/master/src/meltano/core/bundle/discovery.yml#L760), it would read the processed, upcased version

ahhh, that is what needs doing then. I would prefer to uppercase since it follows the default handling of snowflake. What is the lift on a value processor vs a wholesale rework of the target?

Adding a value processor is very easy. If we decide not to have target-snowflake support lowercase

schema

s at all, we would of course need to document that.

Shouldn't it be the responsibility of

target-snowflake

implementation to ensure it changes to uppercase?

@michael_cooper Consulting here first is good if it's not yet clear if it's a bug or a usage issue, but once it's a confirmed bug it's better for the discussion to move to an issue. If we continue to discuss stuff here anyway, we'll have to make sure our findings make it to the issue

will do

@nevin_morgan They're just functions taking in a raw value and returning a processed value, so you'd just need to add a new function that takes a string and upcases it 🙂

I saw that, and I figured out where I skimmed over something. I didn't see the value_processor key at the top of the setting definition class so I was missing that in order to uppercase it I would just do

config:
  value_processor: uppercase

right?

It's a property of a

SettingDefinition

, so it'd need to go here: https://gitlab.com/meltano/meltano/blob/master/src/meltano/core/bundle/discovery.yml#L701

got it. I will get on that shortly and probably put some time to it this weekend then

It's not currently used in

discovery.yml

, but it is here: https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/plugin/singer/tap.py#L73