https://linen.dev logo
Join Slack
Powered by
Hey. Could someone please test tap-stackexchange f...
# troubleshooting
a
Hey. Could someone please test tap-stackexchange from last update 2 days ago? I'm getting same error whatever I try to do (undercut)
I want to load all posts, comments and other data from stack overflow to postgres. First I want to put data in local json via target-jsonl with default settings. Get error singer_sdk.exceptions.FatalAPIError: 400 Client Error: Bad Request for path: /2.3/questions $ meltano config tap-stackexchange list
Copy code
2023-01-28T14:38:24.651576Z [info     ] The default environment 'dev' will be ignored for `meltano config`. To configure a specific environment, please use the option `--environment=<environment name>`.
tags [env: TAP_STACKEXCHANGE_TAGS] current value: ['devops'] (from `meltano.yml`)
	Question tags: Question tags to retrieve.
site [env: TAP_STACKEXCHANGE_SITE] current value: None (default)
	Site: StackExchange site to look for questions.
key [env: TAP_STACKEXCHANGE_KEY] current value: '' (from the TAP_STACKEXCHANGE_KEY variable in `.env`)
	Stack App key: Key to receive a higher request quota.
start_date [env: TAP_STACKEXCHANGE_START_DATE] current value: 1674914400 (from `meltano.yml`)
	Start date: The earliest record date to sync
filter [env: TAP_STACKEXCHANGE_FILTER] current value: None (default)
	Filter: Custom API filter to apply to all requests

To learn more about extractor 'tap-stackexchange' and its settings, visit <https://hub.meltano.com/extractors/tap-stackexchange--meltanolabs>
$ meltano run tap-stackexchange target-jsonl
Copy code
logs in attached file
message has been deleted
$ cat meltano.yml
Copy code
version: 1
default_environment: dev
project_id: acdbe5fb-c92e-4c38-8b4c-ca82197918e2
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-xkcd
    variant: airbyte
    pip_url: git+<https://github.com/MeltanoLabs/tap-airbyte-wrapper.git>
  - name: tap-stackexchange
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/tap-stackexchange.git>
    config:
      tags:
      - devops
      start_date: 1674914400
  loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl
  - name: target-postgres
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/target-postgres.git>
    config:
      sqlaclhemy_url: <postgres+psycopg2://meltano:password@localhost:5431/postgres>
w
Hi @aleksei_razvodov. Using the 
meltano.yml
you provided (and by extension, the latest version of 
tap-stackexchange
) I'm able to run 
meltano run tap-stackexchange target-jsonl
without error. Sorry that this isn't of more help - I just wanted to let you know that the tap is not completely broken, even though it isn't working for you.
a
It worked on new project in new VM. Sorry if someone will face this issue again - I couldn't find the reason
Aaaaaand it doesn't work again. I didn't change anything - ran elt again after 3 hours and it yeilds error at me dark magic
p
I've never used this tap but is it possible that your key is hitting rate limits? I've seen taps that give a similar error when I hit the rate limit, its not the most helpful to only see 
400 Client Error: Bad Request
 vs an explicit rate limit error message if that is the case.
a
I get this error in a one-minute-wide range as well :(
Tap updates last run timestamp even if it fails. And then tries to take a little amount of data
Anyway, where can I configure the rate limit? It's a serious disadvantage if meltano can't run extraction in batches and have to extract all data into local disc/RAM
e
@aleksei_razvodov if you don’t need it, can you try disabling the 
tags
stream:
Copy code
- name: tap-stackexchange
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/tap-stackexchange.git>
    config:
      tags:
      - meltano
    select:
    - !tags.*
Open in Slack
PreviousNext
Powered by