Hey folks I have a mystery involving archiving to S3 using t Meltano #plugins-general

Hey folks, I have a mystery involving archiving to...

connor_lough

07/19/2022, 1:06 AM

Hey folks, I have a mystery involving archiving to S3 using the target-snowflake plugin. I could use your help! In the configuration for target-snowflake, there is a setting called: archive_load_files. This setting is a boolean. However, from its description: "_When incremental replication is used, the archived files will also have the following S3 metadata keys - incremental-key, incremental-key-min and incremental-key-max"._ That this value, archive_load_files, has a value of a dictionary with the above metadata keys inside is represented in the code, here. The problem is, because these key:values are missing, a default of None gets set for 'tap' and the whole job fails. Is this config value supposed to be a boolean? or a dictionary? e.g. archive_load_files: true ~~tap:~~ ~~incremental-key:~~ ~~incremental-key-min:~~ ~~incremental-key-max:~~

edgar_ramirez_mondragon

07/19/2022, 3:59 AM

Hi @connor_lough! Looking at the code that leads to the block you linked to, it seems tat

archive_load_files

is indeed a boolean. The dictionary that’s passed to the method in your link, is populated here: https://github.com/transferwise/pipelinewise-target-snowflake/blob/1cb4fbc2319957549a63df545f3ba84296029f3d/target_snowflake/__init__.py#L286-L305 That block seems to suggest that the

tap

key is sourced from an undocumented setting

tap_id

edgar_ramirez_mondragon

07/19/2022, 4:00 AM

You can check by adding it as

config

meltano.yml

Copy code

- name: target-snowflake
  ...
  config:
    tap_id: test_tap_id

edgar_ramirez_mondragon

07/19/2022, 4:01 AM

If that fixes your problem, then we may need to send a PR to their repo and another one to Meltano Hub to document the setting.

connor_lough

07/19/2022, 4:15 PM

Ah, thank you very much Edgar; that worked! Definitely worth a PR, I've seen a couple of threads that mention this issue w/o fix. Are there docs for submitting a fix? I have only done PRs at my company's GitLab 😮

edgar_ramirez_mondragon

07/20/2022, 2:33 PM

For https://github.com/transferwise/pipelinewise-target-snowflake it’s as simple as 1. fork the repo 2. clone the fork to your machine 3. create a branch to work on your updates (e.g.

document-tap_id

) 4. commit your changes to the README and any other files, and push them to the fork (e.g.

git push -u document-tap_id

) 5. start a PR. The github web ui will suggest that in your fork’s homepage as soon as you push the new branch for Meltano Hub, the process is pretty much the same and the specific file you’ll need to update is https://github.com/meltano/hub/blob/4727b87d21dc532cfaf9216c5d6908620dfd529e/_data/meltano/loaders/target-snowflake/transferwise.yml#L19

edgar_ramirez_mondragon

07/20/2022, 2:34 PM

@connor_lough by all means do submit PRs. I can guarantee at least the Hub one will be quickly reviewed and merged 😄

connor_lough

07/20/2022, 3:20 PM

I most definitely will, as soon as I can get the fix duplicated 😅 Confirmed that the s3 path published properly in s3, then exited my container in an attempt to duplicate. Now it's back to misbehaving

Open in Slack

Previous Next