Hi everyone I could do with some help on exception handling Meltano #singer-tap-development

Hi everyone, I could do with some help on exceptio...

benw-at-birdie

01/21/2022, 11:51 AM

Hi everyone, I could do with some help on exception handling best practices for a custom tap I’m developing: https://github.com/birdiecare/tap-cqc-org-uk The API for the tap returns a status code 429 when it receives too many requests. Following @pablo_seibelt’s fine suggestion I’ve re-implemented

_request_with_backoff

to handle the error. This works perfectly when running meltano in my test environment - the tap pauses for an amount of time before resubmitting the request. However, if I install the tap as a custom extractor using the github repo above then a 429 status code causes the running of the tap exit immediately with the following error 😞 .

Copy code

[error    ] Extraction failed              code=1 job_id=2022-01-21T112740--tap-cqc-org-uk--target-jsonl message=singer_sdk.exceptions.FatalAPIError: 429 Client Error: Too Many Requests for path: /locations/{locationId} name=meltano run_id=ff1f62a6-c0d1-4c6f-8525-7bdd8318001b

Any ideas on what might be causing the difference would be gratefully received 🙏

benw-at-birdie

01/21/2022, 11:51 AM

The meltano.yml config of the tap in the test environment looks like this.

Copy code

- name: tap-cqc-org-uk
    namespace: tap_cqc_org_uk
    executable: ./tap-cqc-org-uk.sh
    capabilities:
    - state
    - catalog
    - discover
    settings:
    - name: start_date
      value: '2022-01-07T00:00:00Z'
    config:
      start_date: '2022-01-07T00:00:00Z'
      partner_code: Birdie Care

The config when installing the extractor from the Github repo looks like this

Copy code

- name: tap-cqc-org-uk
    namespace: tap_cqc_org_uk
    pip_url: git+<https://github.com/birdiecare/tap-cqc-org-uk.git>
    executable: tap-cqc-org-uk
    capabilities:
    - catalog
    - state
    - discover
    settings:
    - name: start_date
      value: '2022-01-07T00:00:00Z'
    config:
      start_date: '2022-01-07T00:00:00Z'
      partner_code: Birdie Care

… where the executable is installed using the command

Copy code

printf 'tap_cqc_org_uk\ngit+<https://github.com/birdiecare/tap-cqc-org-uk.git>\ntap-cqc-org-uk\ncatalogue,state\n\n' | meltano add --custom extractor tap-cqc-org-uk

benw-at-birdie

01/21/2022, 11:53 AM

… and I’m using the following command to invoke the tap in both cases

Copy code

meltano elt tap-cqc-org-uk target-jsonl

visch

01/21/2022, 1:30 PM

There's ways to fix and make this work, but instead I'm going to point you to the SDK's new features for response validaiton https://sdk.meltano.com/en/latest/code_samples.html#custom-response-validation Give this a shot first as it's easier to work with

benw-at-birdie

01/21/2022, 1:46 PM

Thanks Derek, will give it a go 👍

timothy_payne

01/21/2022, 1:59 PM

When it comes to requests sometimes things like this can be caused by inexplicable dependency errors. When testing on your machine are you running in the same virtual environment contained in the poetry build?

benw-at-birdie

01/21/2022, 2:09 PM

I’m not sure to be honest… what’s the best way to check? …just ran

poetry update

and still getting the weird difference in behaviour unfortunately.

benw-at-birdie

01/21/2022, 2:58 PM

Hi @visch unfortunately I’m falling at the first hurdle with https://sdk.meltano.com/en/latest/code_samples.html#custom-response-validation. I can’t seem to import

FatalAPIError

…

poetry.lock

is showing version

0.3.10

of the

singer_sdk

is installed. Is that the right version?

Copy code

ImportError: cannot import name 'FatalAPIError' from 'singer_sdk.exceptions' (/Users/benwhite/Library/Caches/pypoetry/virtualenvs/tap-cqc-org-uk-5_tzXm9--py3.8/lib/python3.8/site-packages/singer_sdk/exceptions.py) name=tap-cqc-org-uk stdio=stderr type=discovery

visch

01/21/2022, 2:59 PM

Poetry manages your dependencies, make sure you're doing that properly here

benw-at-birdie

01/21/2022, 3:09 PM

Ok thanks, any tips for validating the dependencies would be welcome 🙂 . Reading https://sdk.meltano.com/en/latest/python_tips.html#tip-1-intro-to-virtual-environments-poetry-and-pipx but can’t see much other than run

poetry update

edgar_ramirez_mondragon

01/21/2022, 3:31 PM

poetry update

should get you the latest SDK and allow to import the exception

benw-at-birdie

01/24/2022, 11:10 AM

Thanks all. My

pyproject.toml

was out of date - copying the one from the cookiecutter project and re-running

poetry update

did the trick. Now the behaviour of by dev environment is as expected and I’m able to reference the

FatalAPIError

and

RetriableAPIError

exception types 🎉. Apologies if this was an obvious step … I’m new to python development 🙂 . Validating the 429 response and raising a

RetriableAPIError

stops the tap from immediately exiting, thanks Derek 🙌 . The last thing I need to work out is how to adjust the max number of tries and back off factor used. At the moment it gives up after 5 re-tries and doesn’t wait long enough in-between retries for the API to start accepting requests again. Any pointers on how to do this would be very welcome 🙏

benw-at-birdie

01/24/2022, 11:15 AM

… found it https://gitlab.com/meltano/sdk/-/blob/main/singer_sdk/streams/rest.py#L163 😄

benw-at-birdie

01/24/2022, 11:20 AM

Thats worked. Thank you everyone for your help ❤️

edgar_ramirez_mondragon

01/24/2022, 2:07 PM

Awesome @benw-at-birdie!

Open in Slack

Previous Next