https://linen.dev logo
Join Slack
Powered by
:hello: folks, encountered this error when runnin...
# getting-started
k
hello folks, encountered this error when running 
meltano elt tap-hellobaton--inherited target-snowflake --job_id=test
-
Copy code
2022-12-12T20:52:25.205963Z [info     ]   cursor = connection.execute(text( cmd_type=loader name=target-snowflake run_id=cc0d95c8-7985-4bef-a81f-23409f2dc2dc state_id=2022-12-12T205115--tap-hellobaton--inherited--target-snowflake stdio=stderr
2022-12-12T20:52:25.206097Z [info     ] CRITICAL (snowflake.connector.errors.ProgrammingError) 001003 (42000): SQL compilation error: cmd_type=loader name=target-snowflake run_id=cc0d95c8-7985-4bef-a81f-23409f2dc2dc state_id=2022-12-12T205115--tap-hellobaton--inherited--target-snowflake stdio=stderr
2022-12-12T20:52:25.206217Z [info     ] CRITICAL syntax error line 1 at position 14 unexpected '-'. cmd_type=loader name=target-snowflake run_id=cc0d95c8-7985-4bef-a81f-23409f2dc2dc state_id=2022-12-12T205115--tap-hellobaton--inherited--target-snowflake stdio=stderr
2022-12-12T20:52:25.206331Z [info     ] CRITICAL syntax error line 1 at position 14 unexpected '-'. cmd_type=loader name=target-snowflake run_id=cc0d95c8-7985-4bef-a81f-23409f2dc2dc state_id=2022-12-12T205115--tap-hellobaton--inherited--target-snowflake stdio=stderr
2022-12-12T20:52:25.206495Z [info     ] CRITICAL [SQL: MERGE INTO VTS-HELLOBATON.activity USING VTS-HELLOBATON.TMP_ACTIVITY ON VTS-HELLOBATON.activity.id = VTS-HELLOBATON.TMP_ACTIVITY.id WHEN MATCHED THEN UPDATE SET _self = VTS-HELLOBATON.TMP_ACTIVITY._self, actor = VTS-HELLOBATON.TMP_ACTIVITY.actor, child = VTS-HELLOBATON.TMP_ACTIVITY.child, child_type = VTS-HELLOBATON.TMP_ACTIVITY.child_type, created = VTS-HELLOBATON.TMP_ACTIVITY.created, "group" = VTS-HELLOBATON.TMP_ACTIVITY."group", meta = VTS-HELLOBATON.TMP_ACTIVITY.meta, modified = VTS-HELLOBATON.TMP_ACTIVITY.modified, parent = VTS-HELLOBATON.TMP_ACTIVITY.parent, parent_type = VTS-HELLOBATON.TMP_ACTIVITY.parent_type, project = VTS-HELLOBATON.TMP_ACTIVITY.project, type = VTS-HELLOBATON.TMP_ACTIVITY.type, __loaded_at = VTS-HELLOBATON.TMP_ACTIVITY.__loaded_at WHEN NOT MATCHED THEN INSERT (_self, actor, child, child_type, created, "group", id, meta, modified, parent, parent_type, project, type, __loaded_at) VALUES (VTS-HELLOBATON.TMP_ACTIVITY._self, VTS-HELLOBATON.TMP_ACTIVITY.actor, VTS-HELLOBATON.TMP_ACTIVITY.child, VTS-HELLOBATON.TMP_ACTIVITY.child_type, VTS-HELLOBATON.TMP_ACTIVITY.created, VTS-HELLOBATON.TMP_ACTIVITY."group", VTS-HELLOBATON.TMP_ACTIVITY.id, VTS-HELLOBATON.TMP_ACTIVITY.meta, VTS-HELLOBATON.TMP_ACTIVITY.modified, VTS-HELLOBATON.TMP_ACTIVITY.parent, VTS-HELLOBATON.TMP_ACTIVITY.parent_type, VTS-HELLOBATON.TMP_ACTIVITY.project, VTS-HELLOBATON.TMP_ACTIVITY.type, VTS-HELLOBATON.TMP_ACTIVITY.__loaded_at)] cmd_type=loader name=target-snowflake run_id=cc0d95c8-7985-4bef-a81f-23409f2dc2dc state_id=2022-12-12T205115--tap-hellobaton--inherited--target-snowflake stdio=stderr
Anyone have any ideas what this means? It is an inherited plugin i added for hellobaton. Attached my 
meltano.yml
in the thread. Perhaps i need a specific version of sqlite or something?
Copy code
version: 1
send_anonymous_usage_stats: true
project_id: c817018d-de7b-45fb-9fff-260fa84ddaf9
plugins:
  extractors:
  - name: tap-hellobaton
    namespace: tap_hellobaton
    pip_url: git+<https://github.com/dluftspring/tap-hellobaton.git>
    executable: tap-hellobaton
    capabilities:
    - catalog
    - state
    - discover
    settings:
    - name: company
      kind: string
    - name: api_key
      kind: password
    - name: user_agent
      kind: string
    config:
      company: ***
    schema:
      project_attachments:
        created_by:
          type:
          - string
          - 'null'
      time_entries:
        created_by:
          type:
          - string
          - 'null'
        user:
          type:
          - string
          - 'null'
      project_users:
        user:
          type:
          - string
          - 'null'
      milestones:
        start_datetime:
          type:
          - string
          - 'null'
    load_schema: hellobaton
  - name: tap-hellobaton--inherited
    inherit_from: tap-hellobaton
    capabilities:
    - catalog
    - state
    - discover
    config:
      company: ****
      user_agent: Singer Tap for hellobaton
    schema:
      activity:
        group:
          type:
          - string
          - 'null'
    load_schema: ****
cc <!subteam^S02A7AJFXQC>
p
Hey @kevin ! The message 
CRITICAL syntax error line 1 at position 14 unexpected '-'
makes me think that target-snowflake is getting stream names that have invalid characters in them (in this case a hyphen) so its failing. Do those tables exist in Snowflake? What are their names? I assume the hyphen doesnt exist 🤔
Also it probably wont help just yet but we're working on a meltanolabs target-snowflake variant that will be maintained/supported by us https://github.com/MeltanoLabs/target-snowflake. Theres a slack channel #C04BG3978NT where that discussion is happening
k
hey @pat_nadolny! These tables from baton do have some hyphens in them but the tables in snowflake dont currently exist. It is a fresh ingest. snowflake target its ingesting the streams into new schema in an existing snowflake db. How would i combat this?
p
Ideally the target would handle these types of cleanup because theyre source specific, with that said what you can do to get around it is remap the stream names to remove the hyphens and see if that works. If you add this mapper 
meltano add mapper meltano-map-transformer
you can then configure a mapping (stream maps docs) to alias youre streams like this where I rename a stream called test123 to test456:
Copy code
mappers:
  - name: meltano-map-transformer
    variant: meltano
    pip_url: git+<https://github.com/MeltanoLabs/meltano-map-transform.git>
    mappings:
    - name: alias_stream_names
      config:
        stream_maps:
          test123:
            "__alias__": "test456"
Then you can run your normal run command with 
alias_stream_names
mapper in between, something like 
meltano run tap-hellobaton alias_stream_names target-snowflake
Oh I just realized you were using the 
elt
command, I think mappers are only supported with meltano run
k
interesting so i guess theres not mich to do when running the elt command? perhaps change the names on the baton side?
p
actually since the tap is SDK based you should be able to configure those same mapping on the tap side.
Copy code
- name: tap-hellobaton
    variant: dluftspring
    pip_url: git+<https://github.com/dluftspring/tap-hellobaton.git>
    config:
      stream_maps:
        test123:
          "__alias__": "test456"
but you'll have to test that since the tap uses SDK 0.4.x version and the latest is 0.15.0. Should be an easy dependency bump though if you need it upgraded
5 Views
Open in Slack
PreviousNext
Powered by