https://linen.dev logo
Join Slack
Powered by
I’m having an issue with `target-snowflake`. I’m g...
# troubleshooting
d
I’m having an issue with 
target-snowflake
. I’m getting an error when running the 
elt
, but if I fork that repo and add the unmodified version as a custom plugin, it works fine. I’m trying to confirm that I’m not somehow using an older version.
v
could it be the cached catalog issue?
d
@visch Good suggestion, but I don't think so: catalogs are only cached on 
meltano invoke
and 
meltano select
, not `meltano elt`: https://gitlab.com/meltano/meltano/-/issues/2627 🙂
d
It seems to be the way the configuration in 
meltano.yml
is being passed to the plugin. It looks like it’s converted to json, then passed to the plugin as a file: https://github.com/transferwise/pipelinewise-target-snowflake/blob/master/target_snowflake/__init__.py#L429 My 
meltano.yml
looks like:
Copy code
loaders:
  - name: target-snowflake
    variant: transferwise
    pip_url: pipelinewise-target-snowflake==1.12.0
    config:
      account: my-account
      dbname: MY_DATABASE
      user: MY_USER
      warehouse: MY_WAREHOUSE
      file_format: MY_SCHEMA.CSV
      role: MY_ROLE
      schema_mapping:
        public:
          target_schema: RAW_PUBLIC
If I log the file being read in there, I get:
Copy code
{
  "account": "my-account",
  "dbname": "MY_DATABASE",
  "user": "MY_USER",
  "password": "hunter2",
  "warehouse": "MY_WAREHOUSE",
  "file_format": "MY_SCHEMA.CSV",
  "default_target_schema": "TAP_POSTGRES",
  "batch_size_rows": 100000,
  "flush_all_streams": false,
  "parallelism": 0,
  "parallelism_max": 16,
  "schema_mapping": {
    "public.target_schema": "RAW_PUBLIC",
    "public": {
      "target_schema": "RAW_PUBLIC"
    }
  },
  "disable_table_cache": false,
  "add_metadata_columns": false,
  "hard_delete": false,
  "data_flattening_max_level": 0,
  "primary_key_required": true,
  "validate_records": false,
  "no_compression": false,
  "role": "MY_ROLE"
}
This is the bit that’s causing it to fail:
Copy code
"public.target_schema": "RAW_PUBLIC",
Copy code
target-snowflake | Traceback (most recent call last):
target-snowflake |   File "/Users/dean/dev/hi-meltano/.meltano/loaders/target-snowflake/venv/bin/target-snowflake", line 8, in <module>
target-snowflake |     sys.exit(main())
target-snowflake |   File "/Users/dean/dev/hi-meltano/.meltano/loaders/target-snowflake/venv/lib/python3.8/site-packages/target_snowflake/__init__.py", line 438, in main
target-snowflake |     table_cache, file_format_type = get_snowflake_statics(config)
target-snowflake |   File "/Users/dean/dev/hi-meltano/.meltano/loaders/target-snowflake/venv/lib/python3.8/site-packages/target_snowflake/__init__.py", line 79, in get_snowflake_statics
target-snowflake |     table_schemas=stream_utils.get_schema_names_from_config(config))
target-snowflake |   File "/Users/dean/dev/hi-meltano/.meltano/loaders/target-snowflake/venv/lib/python3.8/site-packages/target_snowflake/stream_utils.py", line 43, in get_schema_names_from_config
target-snowflake |     schema_names.append(target.get('target_schema'))
target-snowflake | AttributeError: 'str' object has no attribute 'get'
And now I’ll try with my “custom” version
Copy code
loaders:
  - name: target-snowflake
    pip_url: git+<https://github.com/deanmorin/pipelinewise-target-snowflake.git>
    capabilities:
    - catalog
    - discover
    - state
    config:
      account: my-account
      dbname: MY_DATABASE
      user: MY_USER
      warehouse: MY_WAREHOUSE
      file_format: MY_SCHEMA.CSV
      role: MY_ROLE
      schema_mapping:
        public:
          target_schema: RAW_PUBLIC
Copy code
{
  "account": "my-account",
  "password": "hunter2",
  "role": "MY_ROLE",
  "warehouse": "MY_WAREHOUSE",
  "schema": "TAP_POSTGRES",
  "batch_size": 5000,
  "timestamp_column": "__loaded_at",
  "dbname": "MY_DATABASE",
  "user": "MY_USER",
  "file_format": "MY_SCHEMA.CSV",
  "schema_mapping": {
    "public": {
      "target_schema": "PROVIDER_PUBLIC"
    }
  }
}
It passes in a totally different file to the plugin
d
OK, in the first case you're using 
variant: transferwise
, so it gets all of the default values for settings listed under https://meltano.com/plugins/loaders/snowflake--transferwise.html, so it makes sense it has more properties than the custom plugin you created that doesn't have those settings and defaults
But the 
public.target_schema
key shouldn't be there under 
schema_mapping
, that looks like it could a be a bug 😬 It should only have 
public: {target_schema: "RAW_PUBLIC" }}
, not the dot-separated version
So that's a bug, and I would suggest filing an issue: https://gitlab.com/meltano/meltano/-/issues/new
But I think we can work around the issue: Do you actually need 
schema_mapping
? Or could you set 
default_target_schema
?
d
My “custom” plugin is just a fork of the transferwise variant
At least I think it is, lemme double check
d
@dean_morin I'm aware it's just a fork, but unless the definition has 
variant: transferwise
, it's not going to know what settings it supports and what default values it should use
d
Yes it is, the transferwise variant is 
pipeline-target-snowflake
: https://meltano.com/plugins/loaders/snowflake--transferwise.html
Oh gotcha
d
Either way, I think you can work around the issue by removing 
schema_mapping
and adding 
default_target_schema: RAW_PUBLIC
d
I’m going to be mapping a few more schemas so I think I need that set, but I think I found a potential other workaround. If I add it as a custom plugin (just change 
pip_url
from my fork to the regular one), then it works.
So for any of the “built-in” extractors & loaders, it sounds like 
meltano
provide defaults that aren’t necessarily the defaults for the singer tap/target?
d
Would you mind filing an issue for that?
d
Will do!
d
Thanks!
d
Open in Slack
PreviousNext
Powered by