Hello I am getting this error while doing elt command.

I am getting this error
ELT could not be completed: Cannot start extractor: Catalog discovery failed: command ['/Users/mayankarora/code/my-meltano-project/.meltano/extractors/tap-bigquery/venv/bin/tap-bigquery'
[2:49 PM] I don't know why this error is coming\

@mayank_arora can you share more of your error log? Usually further up in the logs it explains why the discovery failed i.e. authentication, connection, etc.

Also I see you posted a similar message in #CMN8HELB0 would you mind deleting that one so theres not two threads going? I dont want anyone wasting their time answering if we're already discussing in this thread 😄

Yes sure

Can you try running the tap like this? It will probably give you better error messages

meltano --log-level=debug invoke tap-bigquery

From your logs I didnt see anything other than the fact that it cant run those SQL queries. It might be useful to try running one of those in your database client to see if there is something wrong with the syntax maybe

This is the error I am getting now

It looks like the tap is expecting a streams setting https://hub.meltano.com/extractors/tap-bigquery#streams-setting in your config but it’s not there. Can you check your meltano.yml file? Also running “meltano config tap-bigquery” will print your current t config to help debug

Yes it prints the below config

{
  "credentials_path": "bigquery-credentials/google-service-account-mayank-key.json",
  "start_datetime": "2021-07-21T00:00:00Z",
  "start_always_inclusive": true
}

@pat_nadolny Do you know what might be happening there?

Yes it prints the below config

@mayank_arora were you able to add the streams setting and see it show up in your config?

earlier I was using meltano 1.99.0 and with this version, everything was fine

I'm not sure what would have changed related to this 🤔 Were you able to try my suggestion of using

invoke

in debug mode? Any additional details? Also just to confirm, you were able to run this SQL in your SQL client with no issues?

SELECT event_type,`time`,user_id,event_properties,user_properties,`groups`,insert_id,inserted_at FROM root-station-198710.dbt_out_zone_amplitude.ecommerce_report_viewed WHERE 1=1 AND datetime '2023-07-21 00:00:00.000000' <= CAST(inserted_at as datetime) AND CAST(inserted_at as datetime) < datetime '2023-08-05 12:44:55.923261' ORDER BY inserted_at LIMIT 100

In my earlier version, I never use streams because it's not required at that time. Yes, I used invoke but it generates the same error.

It looks like https://meltano.slack.com/archives/C01TCRBBJD7/p1690539313030839?thread_ts=1690465513.157009&cid=C01TCRBBJD7 is the invoke logs but its missing the required streams config. Then https://meltano.slack.com/archives/C01TCRBBJD7/p1691240342032159?thread_ts=1690465513.157009&cid=C01TCRBBJD7 is the logs from a run or elt command.

The issue is: After updating the new version of meltano, I was getting the error of

ELT could not be completed: Cannot start extractor: Catalog discovery failed:

when I ran the ELT command, then after I debug and found the issue and shared the logs with you. Because of this issue I was not able to run the elt command. same meltano.yml file I was using it without stream in version 1.99.0, the issue comes when I updated the meltano version.