Hi Team, I am new to Meltano and am trying to set ...
# troubleshooting
y
Hi Team, I am new to Meltano and am trying to set up tap-snowflake. The setting can discover the schema and return a copy of tab.properties.json correctly. The issue no record is returned from running
meltano elt tap-snowflake target-jsonl
I follow all the steps from this thread(https://meltano.slack.com/archives/C01TCRBBJD7/p1621519111034600), but it will return any record
Copy code
- name: tap-snowflake
    variant: transferwise
    pip_url: pipelinewise-tap-snowflake
    capabilities:
    - catalog
    - discover
    config:
      account: test
      dbname: TEST1
      role: test
      user: test
      warehouse: test
      password: $SNOWFLAKE_PASSWORD
      tables: TEST1.AB_TESTING.RELATED_PARTY_OPPORTUNITY
    # catalog: extract/tap-snowflake-related-party.properties.json
    metadata:
      TEST1-AB_TESTING-RELATED_PARTY_OPPORTUNITY:
        replication-method: FULL_TABLE
    select:
    - TEST1.AB_TESTING.RELATED_PARTY_OPPORTUNITY*.*
    - TEST1-AB_TESTING-RELATED_PARTY_OPPORTUNITY*.*
    - RELATED_PARTY_OPPORTUNITY*.*
Below is the output. I add the LOGG.info in the get_stream function which return resovle_catelog result {‘streams’: []}
Copy code
2022-06-24T14:26:18.150996Z [info     ] time=2022-06-24 14:26:18 name=tap_snowflake level=INFO message=resovle_catelog result {'streams': []} cmd_type=extractor name=tap-snowflake run_id=1c2d76e8-9a1c-4c37-b336-c935477e6c87 state_id=2022-06-24T142611--tap-snowflake--target-jsonl stdio=stderr
2022-06-24T14:26:18.151827Z [debug    ] {"type": "STATE", "value": {"currently_syncing": null}} cmd_type=extractor name=tap-snowflake (out) run_id=1c2d76e8-9a1c-4c37-b336-c935477e6c87 state_id=2022-06-24T142611--tap-snowflake--target-jsonl stdio=stdout
2022-06-24T14:26:18.315641Z [debug    ] {"currently_syncing": null}    cmd_type=loader name=target-jsonl (out) run_id=1c2d76e8-9a1c-4c37-b336-c935477e6c87 state_id=2022-06-24T142611--tap-snowflake--target-jsonl stdio=stdout
2022-06-24T14:26:18.339183Z [info     ] Incremental state has been updated at 2022-06-24 14:26:18.339055.
2022-06-24T14:26:18.340450Z [debug    ] Incremental state: {'currently_syncing': None}
2022-06-24T14:26:18.345291Z [debug    ] Deleted configuration at /project/.meltano/run/elt/2022-06-24T142611--tap-snowflake--target-jsonl/1c2d76e8-9a1c-4c37-b336-c935477e6c87/target.c6523ca2-3ebe-464b-84d0-2fd6f79740ac.config.json
2022-06-24T14:26:18.347285Z [debug    ] Deleted configuration at /project/.meltano/run/elt/2022-06-24T142611--tap-snowflake--target-jsonl/1c2d76e8-9a1c-4c37-b336-c935477e6c87/tap.fa14a899-fb00-4797-96d9-79e7933f41e2.config.json
2022-06-24T14:26:18.348266Z [info     ] Extract & load complete!       name=meltano run_id=1c2d76e8-9a1c-4c37-b336-c935477e6c87 state_id=2022-06-24T142611--tap-snowflake--target-jsonl
2022-06-24T14:26:18.349153Z [info     ] Transformation skipped.        name=meltano run_id=1c2d76e8-9a1c-4c37-b336-c935477e6c87 state_id=2022-06-24T142611--tap-snowflake--target-jsonl
p
@yijie_wang I set mine up a while ago so its not super fresh in my mind but I'm curious if the dashes in your table references are causing issues, if its helpful I have mine configured in https://github.com/meltano/squared/blob/9cf437c4ec811c850f37a1f9bb69911487e9e836/data/extract/extractors.meltano.yml#L103 and environment level configs at https://github.com/meltano/squared/blob/9cf437c4ec811c850f37a1f9bb69911487e9e836/data/environments/userdev.meltano.yml#L16. Maybe try using
*
for your metadata selection on one of the tables and see if that gets some data flowing. Also maybe the first
*
as part of your
select
criteria
TEST1.AB_TESTING.RELATED_PARTY_OPPORTUNITY*.*
->
TEST1.AB_TESTING.RELATED_PARTY_OPPORTUNITY.*
. I dont know but taking a few guesses at things I'd try 😄
y
@pat_nadolny thank you so much for your help. After a couple of tries, the following syntax worked!
Copy code
select:
       - TEST1-AB_TESTING-RELATED_PARTY_OPPORTUNITY.*
This represents
Copy code
select:
   - <"tap_stream_id">-<table_name>.*