hey all, didnt ever have to do this before (i dont recall) but we now need the right auth to install the snowflake target? Would I have to Generate a Personal Access Token on GitLab ?

(meltano_env) ubuntu@ip-172-31-26-110:~/repos/lane-meltano$ meltano add loader target-snowflake
Loader 'target-snowflake' already exists in your Meltano project
To add it to your project another time so that each can be configured differently,
add a new plugin inheriting from the existing one with its own unique name:
	meltano add loader target-snowflake--new --inherit-from target-snowflake

Installing loader 'target-snowflake'...
Username for '<https://gitlab.com>': 
Password for '<https://gitlab.com>': 
Loader 'target-snowflake' could not be installed: failed to install plugin 'target-snowflake'.
  Running command git clone --filter=blob:none --quiet <https://gitlab.com/gitlab-data/edcast-target-snowflake> /tmp/pip-req-build-hdft4rrz
  remote: HTTP Basic: Access denied. The provided password or token is incorrect or your account has 2FA enabled and you must use a personal access token instead of a password. See <https://gitlab.com/help/topics/git/troubleshooting_git#error-on-git-fetch-http-basic-access-denied>
  fatal: Authentication failed for '<https://gitlab.com/gitlab-data/edcast-target-snowflake.git/>'
  error: subprocess-exited-with-error
  
  × git clone --filter=blob:none --quiet <https://gitlab.com/gitlab-data/edcast-target-snowflake> /tmp/pip-req-build-hdft4rrz did not run successfully.
  │ exit code: 128
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× git clone --filter=blob:none --quiet <https://gitlab.com/gitlab-data/edcast-target-snowflake> /tmp/pip-req-build-hdft4rrz did not run successfully.
│ exit code: 128
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Failed to install plugin(s)

If the repo is public, you shouldn't need to use any credentials afaik. Which makes think they might've made the repo private. cc @pat_nadolny @taylor

I don’t have access to that repo either, I wonder if it was deleted or if the pip url is wrong. If there’s no particular reason to use that specific target variant I would suggest removing it from your project then readding the default target snowflake variant listed on the hub

@jstark did you all make that repo private?

hmmm alright i think it is private. What pip url should i switch it to??

the default target-snowflake works quite well https://hub.meltano.com/loaders/target-snowflake

cool ill give that a try @taylor

for that one the pip_url is

pipelinewise-target-snowflake

but the config might be different. It might be easier to remove that loader and do a fresh install of the default variant

yeah the config might be different. Just do a

pip install pipelinewise-target-snowflake

?

I would recommend doing

meltano remove loader target-snowflake

followed by

meltano add loader target-snowflake

which will do everything you want. replacing just the pip_url may have some unintended consequences

thanks @taylor!

Hey hi @kevin, the new URL for

target-snowflake--edcast

is: https://gitlab.com/gitlab-data/target-snowflake-edcast Sorry about that, I do not have the info when/why this was changed, can look it up or discuss it with the team.

hmm using

pipelinewise-target-snowflake

and running the elt command i do get this error:

snowflake.connector.errors.ProgrammingError: 090105 (22000): Cannot perform CREATE TABLE. This session does not have a current database. Call 'USE DATABASE', or use a qualified name

I have granted the role and user usage and other grants on the appropriate db and schema. Meltano yml looks like:

- name: target-snowflake
    variant: transferwise
    pip_url: pipelinewise-target-snowflake
    config:
      account: **************
      dbname: RAW_MELTANO_DB
      user: meltano_user
      warehouse: ELT_WH
      role: meltano_role
      file_format: csv
      default_target_schema: HELLOBATON

@kevin were you able to run the grants in https://github.com/transferwise/pipelinewise-target-snowflake#pre-requirements ? It’s possible it’s a permissions hting

hey @taylor I was able to run all the commands. The permissions seem right to me:

did that seem to fix it? The only other thread I found with that problem is https://meltano.slack.com/archives/C01TCRBBJD7/p1648060380357579 which pointed to a USAGE grant. based on the error you’re seeing it has to be something on the snowflake permissions side

The target repo has a bit more detail on setting up a role with proper permissions https://github.com/transferwise/pipelinewise-target-snowflake#pre-requirements

thanks @pat_nadolny! i successfully ran through all the permissioning steps to no avail but perhaps i am missing something. Will continue to debug

Then maybe it makes sense to use the target-snowflake indeed as @taylor already proposed @kevin, I am currently also using that one as I had a similar issue with the custom tap.

yeah got past that! when running

meltano elt tap-hellobaton target-snowflake

i get:

2023-04-03T14:37:59.796720Z [info     ] snowflake.connector.errors.ProgrammingError: 000904 (42000): SQL compilation error: error line 1 at position 767 cmd_type=loader name=target-snowflake run_id=e7307e7e-7ee4-44b6-83f4-b658ff3e6205 state_id=2023-04-03T143457--tap-hellobaton--target-snowflake stdio=stderr
2023-04-03T14:37:59.796877Z [info     ] invalid identifier 'GROUP'

Looks like it is complaining about a column named

group

renaming the column is not an option though and it comes directly from the json of the connector. Any ideas on how to bypass this?

@kevin can you say more about where you're at now? Are you trying to write to an existing table or let the target create the table for you? You said you got past the permissions piece, do you see tables and records in snowflake now? How far does it get before the error?

yeah i am trying to write to an existing table (

raw_meltano_db.hellobaton

). All the column names are there where it was extracted using the old target snowflake in a previous pipeline:

git+<https://gitlab.com/gitlab-data/edcast-target-snowflake>

Since that old pip url changed, the new pip url caused the import errors i mentioned above (

cryptography

) . That is when i switched over to the

pipelinewise-target-snowflake

target and now when running

meltano elt tap-hellobaton target-snowflake

I started getting the most recent error

Thanks for the update - you can also try using stream maps to rename the column(s) so you dont need to maintain a fork for minor changes like that. I'm still curious what that error is saying though because I think theoretically the target should mutate your table if it receives an updated schema i.e. automatically add that group column

cool thanks for the doc, could we also create a python script that renames the column name inside of a mappers directory and reference it in meltano ym like so?:

taps:
  - name: tap-hellobaton
    # ... (other tap configuration)
    mappers:
      - rename_columns_mapper

I'm sure thats possible somehow but I dont know exactly how. Heres an example of how I change the property name of one of my streams from

TITLE

to

title

using stream maps on the SDK based target, if its helpful. You'd configure it on the tap side though

noted i see it here:

- name: target-apprise-singer-activity
        config:
          stream_maps:
            PROD-SLACK_NOTIFICATIONS-SLACK_ALERTS:
              title: TITLE
              body: BODY

should i rename

PROD-SLACK_NOTIFICATIONS-SLACK_ALERTS

to activity t~ap-hellobaton~

should i rename

PROD-SLACK_NOTIFICATIONS-SLACK_ALERTS

to activity

yep!

cool i still get the same error but my meltano yml file looks like:

version: 1
send_anonymous_usage_stats: true
project_id: c817018d-de7b-45fb-9fff-260fa84ddaf9
plugins:
  extractors:
  - name: tap-hellobaton
    namespace: tap_hellobaton
    pip_url: git+<https://github.com/dluftspring/tap-hellobaton.git>
    executable: tap-hellobaton
    capabilities:
    - catalog
    - state
    - discover
    settings:
    - name: company
      kind: string
    - name: api_key
      kind: password
    - name: user_agent
      kind: string
    config:
      company: ***
      user_agent: Singer Tap for hellobaton
    load_schema: hellobaton
  loaders:
  - name: target-snowflake
    variant: transferwise
    pip_url: pipelinewise-target-snowflake
    config:
      account: *********
      dbname: RAW_MELTANO_DB
      user: meltano_user
      password: ******
      warehouse: ELT_WH
      role: meltano_role
      file_format: RAW_MELTANO_DB.HELLOBATON.CSV
    metadata:
      column_mapping:
        GROUP: user_group
  mappers:
  - name: meltano-map-transformer
    variant: meltano
    pip_url: git+<https://github.com/MeltanoLabs/meltano-map-transform.git>
    executable: meltano-map-transform
    mappings:
    - name: column_rename
      config:
        stream_maps:
          activity:
            group: activity_group

Am i missing anything? I did a fresh meltano install

You may have to set add_metadata_columns to true

ah so

add_metadata_columns: true

?

Yeah, for existing rows they’ll be null but they should be set for new/merged ones

thanks @edgar_ramirez_mondragon! i got one more quick question so i do get this error now:

2023-04-03T19:57:51.474009Z [info     ]     if 'string' in schema['properties'][key]['type'] and \ cmd_type=loader name=target-snowflake run_id=aa51541d-d597-4821-b33d-ea733638e4f6 state_id=2023-04-03T193859--tap-hellobaton--target-snowflake stdio=stderr
2023-04-03T19:57:51.474168Z [info     ] KeyError: 'type'               cmd_type=loader name=target-snowflake run_id=aa51541d-d597-4821-b33d-ea733638e4f6 state_id=2023-04-03T193859--tap-hellobaton--target-snowflake stdio=stderr
2023-04-03T19:57:51.816177Z [error    ] Loading failed

is this a case where we need to override the schema under the appropriate stream?

So what’s your latest

meltano.yml

?