https://linen.dev logo
Join Slack
Powered by
Hi, I am trying to configure `tap-snowflake` , wit...
# singer-taps
v
Hi, I am trying to configure 
tap-snowflake
, with a config like this --
Copy code
plugins:
  extractors:
  - name: tap-snowflake
    variant: meltanolabs
    pip_url: meltanolabs-tap-snowflake
    config:
      account: XXX.eu-west-1
      database: TEST
      role: XXX_PROD
      schema: TORNADO
      tables: 
        - TEST.TORNADO.AUDIT
      user: XXX
      warehouse: MICRO_SERVICES_WH
The table TEST.TORNADO.AUDIT table exists and can view data from it while logged into this user with this role. I can't connect to this tap and extract data however. This is the error --
Copy code
> meltano config tap-snowflake test

2024-08-09T12:48:57.007182Z [info     ] The default environment 'dev' will be ignored for `meltano config`. To configure a specific environment, please use the option `--environment=<environment name>`.
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Plugin configuration is invalid
No RECORD or BATCH message received
Can someone help understand what is going wrong here?
v
Looks like you're missing a select statement to me
Personally I run 
meltano invoke tap-snowflake
to see if something works and just ctrl+c it if data starts coming through. The test function works as it answers that something is wrong, but you have to dive into why
e
I've put up an issue to improve the UX of this command: https://github.com/meltano/meltano/issues/8703
v
Can you please guide me to some documentation how to add the 
select
statement in the configuration?
I tried 
meltano invoke tap-snowflake
and got this output
Copy code
> meltano invoke tap-snowflake       
2024-08-12T07:53:51.019786Z [info     ] Environment 'dev' is active   
/home/vibhav/source/learn/meltano-project/test/.meltano/extractors/tap-snowflake/venv/lib/python3.11/site-packages/snowflake/sqlalchemy/base.py:1068: SAWarning: The GenericFunction 'flatten' is already registered and is going to be overridden.
  functions.register_function("flatten", flatten)
2024-08-12 13:23:52,071 | INFO     | snowflake.connector.connection | Snowflake Connector for Python Version: 3.12.0, Python Version: 3.11.2, Platform: Linux-5.15.153.1-microsoft-standard-WSL2-x86_64-with-glibc2.36
2024-08-12 13:23:52,072 | INFO     | snowflake.connector.connection | Connecting to GLOBAL Snowflake domain
2024-08-12 13:23:52,072 | INFO     | snowflake.connector.connection | This connection is in OCSP Fail Open Mode. TLS Certificates would be checked for validity and revocation status. Any other Certificate Revocation related exceptions or OCSP Responder failures would be disregarded in favor of connectivity.
2024-08-12 13:23:53,065 | INFO     | snowflake.connector.cursor | Number of results in first chunk: 1
2024-08-12 13:23:53,240 | INFO     | snowflake.connector.cursor | Number of results in first chunk: 1
2024-08-12 13:23:53,415 | INFO     | snowflake.connector.cursor | Number of results in first chunk: 1
2024-08-12 13:23:53,660 | INFO     | snowflake.connector.cursor | Number of results in first chunk: 26
2024-08-12 13:23:53,841 | INFO     | snowflake.connector.cursor | Number of results in first chunk: 1
2024-08-12 13:23:54,022 | INFO     | snowflake.connector.cursor | Number of results in first chunk: 1
2024-08-12 13:23:54,173 | INFO     | snowflake.connector.cursor | Number of results in first chunk: 1
2024-08-12 13:23:54,364 | INFO     | snowflake.connector.cursor | Number of results in first chunk: 0
2024-08-12 13:23:55,527 | INFO     | snowflake.connector.cursor | Number of results in first chunk: 411
...a lot of chunks in between...
2024-08-12 13:24:13,931 | INFO     | snowflake.connector.cursor | Number of results in first chunk: 1
2024-08-12 13:24:14,087 | INFO     | snowflake.connector.cursor | Number of results in first chunk: 1
{"type": "STATE", "value": {}}
2024-08-12 13:24:14,088 | INFO     | snowflake.connector.connection | closed
2024-08-12 13:24:14,088 | INFO     | snowflake.connector.connection | No async queries seem to be running, deleting session
I've added a select clause in the plugin, but still unsure what exactly I need to select
Copy code
plugins:
  extractors:
  - name: tap-snowflake
    variant: meltanolabs
    pip_url: meltanolabs-tap-snowflake
    config:
      account: XXX.eu-west-1
      database: TEST
      role: XXX_PROD
      schema: TORNADO
      tables: 
        - TEST.TORNADO.AUDIT
      user: XXX
      warehouse: MICRO_SERVICES_WH
    select:
      - tables.*
I am unsure what the stream name should be
Copy code
[warning  ] Stream `tables` was not found in the catalog
r
You can run
Copy code
meltano select tap-snowflake --list --all
to see what streams/properties are available.
👍 1
272 Views
Open in Slack
PreviousNext
Powered by