https://linen.dev logo
Join Slack
Powered by
Hey team! I am looking at making a custom extract...
# troubleshooting
j
Hey team! I am looking at making a custom extractor for Afterpay and running into a status code error 412, which is
Copy code
412	invalid_code	Will be returned if the code: has expired, has been used, is invalid.
I am having trouble finding where to re-authenticator my auth function. Can anyone help me? Currently in my client.py
Copy code
@property
    @cached
    def authenticator(self) -> afterpayAuthenticator:
        """Return a new authenticator object."""
        return afterpayAuthenticator.create_for_stream(self)
c
Are you saying that the authentication mechanism used by Afterpay will return a field with error code 
412
somewhere in the response body and you need to handle the situation? Are you writing a custom Authenticator class with the Meltano SDK?
j
Correct and correct. Using the custom extractor tutorial with some modifications.
c
j
Thanks. I will take a look
What I mean is that I am able to stream data with the initial auth, but after about 1000 requests, the access token is expired and will need a new access token.
e
@justin_fung and that is independent of the 
expires_in
value in the auth response? Meaning, even if 
expires_in
is still far in the future with a value like 86399, after 1000 requests you always have to re-authenticate?
j
Yes, but I wonder if 86399 would be in ms or seconds because if it is ms, 86 seconds (after converting from ms) feels about right when it breaks.
e
The field 
expires_in
is the time (in seconds) that the token is valid from when its generated. The token is not usable after this time elapses.
I think it's in seconds: https://developers.afterpay.com/docs/api/online-api%2Fapi-architecture%2Fauthentication#access-token-response
j
Interesting. Just ran my extractor and it ran from 
19:14:50
to 
19:15:31
and got a total of 3100.
There is a total of 10000 pages of 20 records, so it went through over 1/10 of the total.
Do you want me to post the full error?
e
Yeah, that might help
j
```2023-11-15T191530.540622Z [info ] 2023-11-15 121530,534 | INFO | singer_sdk.metrics | METRIC: {"type": "timer", "metric": "sync_duration", "value": 38.543965339660645, "tags": {"stream": "payments", "context": {}, "status": "failed"}} cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay 2023-11-15T191530.541650Z [info ] 2023-11-15 121530,534 | INFO | singer_sdk.metrics | METRIC: {"type": "counter", "metric": "record_count", "value": 3100, "tags": {"stream": "payments", "context": {}}} cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay 2023-11-15T191530.628482Z [info ] 2023-11-15 121530,534 | ERROR | tap-afterpay | An unhandled error occurred while syncing 'payments' cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay 2023-11-15T191530.630003Z [info ] Traceback (most recent call last): cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay 2023-11-15T191530.632606Z [info ] File "C:\Users\Ruggable\Documents\ruggable\Ruggable-GitHub\tap-afterpay\.meltano\extractors\tap-afterpay\venv\lib\site-packages\singer_sdk\streams\core.py", line 1187, in sync cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay 2023-11-15T191530.634581Z [info ] for _ in self._sync_records(context=context): cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay 2023-11-15T191530.636604Z [info ] File "C:\Users\Ruggable\Documents\ruggable\Ruggable-GitHub\tap-afterpay\.meltano\extractors\tap-afterpay\venv\lib\site-packages\singer_sdk\streams\core.py", line 1081, in _sync_records cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay 2023-11-15T191530.637606Z [info ] for record_result in self.get_records(current_context): cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay 2023-11-15T191530.640603Z [info ] File "C:\Users\Ruggable\Documents\ruggable\Ruggable-GitHub\tap-afterpay\.meltano\extractors\tap-afterpay\venv\lib\site-packages\singer_sdk\streams\rest.py", line 574, in get_records cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay 2023-11-15T191530.642627Z [info ] for record in self.request_records(context): cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay 2023-11-15T191530.644626Z [info ] File "C:\Users\Ruggable\Documents\ruggable\Ruggable-GitHub\tap-afterpay\.meltano\extractors\tap-afterpay\venv\lib\site-packages\singer_sdk\streams\rest.py", line 395, in request_records cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay 2023-11-15T191530.646632Z [info ] resp = decorated_request(prepared_request, context) cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay 2023-11-15T191530.648020Z [info ] File "C:\Users\Ruggable\Documents\ruggable\Ruggable-GitHub\tap-afterpay\.meltano\extractors\tap-afterpay\venv\lib\site-packages\backoff\_sync.py", line 105, in retry cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay 2023-11-15T191530.650082Z [info ] ret = target(*args, **kwargs) cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay 2023-11-15T191530.651225Z [info ] File "C:\Users\Ruggable\Documents\ruggable\Ruggable-GitHub\tap-afterpay\.meltano\extractors\tap-afterpay\venv\lib\site-packages\singer_sdk\streams\rest.py", line 274, in _request cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay 2023-11-15T191530.653252Z [info ] self.validate_response(response) cmd_type=elb…
``` 2023-11-15T191530.711436Z [info ] return __callback(*args, **kwargs) cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay 2023-11-15T191530.712436Z [info ] File "C:\Users\Ruggable\Documents\ruggable\Ruggable-GitHub\tap-afterpay\.meltano\extractors\tap-afterpay\venv\lib\site-packages\singer_sdk\tap_base.py", line 501, in invoke cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay 2023-11-15T191530.734734Z [info ] tap.sync_all() cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay 2023-11-15T191530.735775Z [info ] File "C:\Users\Ruggable\Documents\ruggable\Ruggable-GitHub\tap-afterpay\.meltano\extractors\tap-afterpay\venv\lib\site-packages\singer_sdk\tap_base.py", line 460, in sync_all cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay 2023-11-15T191530.737790Z [info ] stream.sync() cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay 2023-11-15T191530.739801Z [info ] File "C:\Users\Ruggable\Documents\ruggable\Ruggable-GitHub\tap-afterpay\.meltano\extractors\tap-afterpay\venv\lib\site-packages\singer_sdk\streams\core.py", line 1194, in sync cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay 2023-11-15T191530.742373Z [info ] raise ex cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay 2023-11-15T191530.743349Z [info ] File "C:\Users\Ruggable\Documents\ruggable\Ruggable-GitHub\tap-afterpay\.meltano\extractors\tap-afterpay\venv\lib\site-packages\singer_sdk\streams\core.py", line 1187, in sync cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay 2023-11-15T191530.746334Z [info ] for _ in self._sync_records(context=context): cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay 2023-11-15T191530.748346Z [info ] File "C:\Users\Ruggable\Documents\ruggable\Ruggable-GitHub\tap-afterpay\.meltano\extractors\tap-afterpay\venv\lib\site-packages\singer_sdk\streams\core.py", line 1081, in _sync_records cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay 2023-11-15T191530.750347Z [info ] for record_result in self.get_records(current_context): cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay 2023-11-15T191530.753203Z [info ] File "C:\Users\Ruggable\Documents\ruggable\Ruggable-GitHub\tap-afterpay\.meltano\extractors\tap-afterpay\venv\lib\site-packages\singer_sdk\streams\rest.py", line 574, in get_records cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay 2023-11-15T191530.754197Z [info ] for record in self.request_records(context): cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay 2023-11-15T191530.756747Z [info ] File "C:\Users\Ruggable\Documents\ruggable\Ruggable-GitHub\tap-afterpay\.meltano\extractors\tap-afterpay\venv\lib\site-packages\singer_sdk\streams\rest.py", line 395, in request_records cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay 2023-11-15T191530.758749Z [info ] resp = decorated_request(prepared_request, context) cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay 2023-11-15T191530.760745Z [info ] File "C:\Users\Ruggable\Documents\ruggable\Ruggable-GitHub\tap-afterpay\.meltano\extractors\tap-afterpay\venv\lib\site-packages\backoff\_sync.py", line 105, in retry cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay 2023-11-15T191530.762459Z [info …
Before that I am getting pretty standard stream responses:
Copy code
2023-11-15T19:15:30.534958Z [info     ] 2023-11-15 12:15:30,533 | INFO     | singer_sdk.metrics   | METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.140062, "tags": {"stream": "payments", "endpoint": "/v2/payments", "http_status_code": 412, "status": "failed"}} cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay
2023-11-15T19:15:30.537971Z [info     ] 2023-11-15 12:15:30,534 | INFO     | singer_sdk.metrics   | METRIC: {"type": "counter", "metric": "http_request_count", "value": 155, "tags": {"stream": "payments", "endpoint": "/v2/payments"}} cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay
e
Ok, what does you authenticator class look like?
j
Copy code
"""afterpay Authentication."""

from __future__ import annotations

from singer_sdk.authenticators import OAuthAuthenticator, SingletonMeta


# The SingletonMeta metaclass makes your streams reuse the same authenticator instance.
# If this behaviour interferes with your use-case, you can remove the metaclass.
class afterpayAuthenticator(OAuthAuthenticator, metaclass=SingletonMeta):
    """Authenticator class for AfterPay."""

    @property
    def oauth_request_body(self) -> dict:
        """Define the OAuth request body for the AfterPay API."""
        return {
            "client_id": self.config["client_id"],
            "client_secret": self.config["client_secret"],
            "scope": "merchant_api_v2",
            "grant_type": "client_credentials",
        }

    @classmethod
    def create_for_stream(cls, stream) -> afterpayAuthenticator:
        """Instantiate an authenticator for a specific Singer stream."""
        return cls(
            stream=stream,
            auth_endpoint="<https://merchant-auth.afterpay.com/v2/oauth2/token>",
        )
e
Ok, that looks alright. This is a pattern I was not familiar with (an API requiring re-auth after a number of requests). You can try overriding validate_response:
Copy code
from http import HTTPStatus
from singer_sdk.exceptions import RetriableAPIError

class MyStreamClass(...):
    extra_retry_statuses = [HTTPStatus.PRECONDITION_FAILED, HTTPStatus.TOO_MANY_REQUESTS]

    def validate_response(self, response):
        # Requires re-authentication
        if respone.status_code == HTTPStatus.PRECONDITION_FAILED:
            self.authenticator = afterpayAuthenticator.create_for_stream(self)
        super().validate_response(response)
j
Just making sure. Would this be inside the client.py?
e
Yeah, this would be in your "client" base stream class
j
ran into this:
Copy code
2023-11-15T21:28:35.076002Z [info     ] AttributeError: can't set attribute 'authenticator' cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay
Ah one sec
Never mind. Ran into the same error.
e
Hmm, that 
@cached
is ok but it kinda gets in the way. Can you use 
functools.cached_property
instead?
Copy code
from http import HTTPStatus
from functools import cached_property
from singer_sdk.exceptions import RetriableAPIError

class MyStreamClass(...):
    extra_retry_statuses = [HTTPStatus.PRECONDITION_FAILED, HTTPStatus.TOO_MANY_REQUESTS]

    @cached_property
    def authenticator(self) -> afterpayAuthenticator:
        """Return a new authenticator object."""
        return afterpayAuthenticator.create_for_stream(self)

    def validate_response(self, response):
        # Requires re-authentication
        if respone.status_code == HTTPStatus.PRECONDITION_FAILED:
            self.authenticator = afterpayAuthenticator.create_for_stream(self)
        super().validate_response(response)
Note that this will require Python 3.8+
j
Looks like it is working!
I did get this response for a little bit:
Copy code
2023-11-15T21:48:48.717633Z [info     ] 2023-11-15 14:48:48,717 | ERROR    | root                 | Backing off 16.35 seconds after 4 tries calling function <bound method RESTStream._request of <tap_afterpay.streams.PaymentsStream object at 0x0000019A92D0DA80>> with args (<PreparedRequest [GET]>, None) and kwargs {} cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay
But went back to normal.
It did look like it ended up breaking with the same error code... Give me a minute to dig into it
I am trying to restart the process (meltano run) and running into this error:
Copy code
2023-11-15T21:48:48.717633Z [info     ] 2023-11-15 14:48:48,717 | ERROR    | root                 | Backing off 16.35 seconds after 4 tries calling function <bound method RESTStream._request of <tap_afterpay.streams.PaymentsStream object at 0x0000019A92D0DA80>> with args (<PreparedRequest [GET]>, None) and kwargs {} cmd_type=elb consumer=False name=tap-afterpay producer=True stdio=stderr string_id=tap-afterpay
e
Ok, can you remove the metaclass from your authenticator? It might cause more re-authentications than necessary but it's worth trying.
j
It errored out immediately.
I believe I was able to stream all the needed data. I re-ran it and it errored out at 24000 records and like I said before, we have about 10000 pages (per the response) and 20 records per page.
My only other question would be: Would this error stop and remove all of the data from the streaming process to our database and if it does, how could I handle this error?
e
No, it wouldn't remove any data. My suggestion above should handle the error and force a re-auth and retry if 
412
is encountered.
j
@edgar_ramirez_mondragon Would you be available to help review my pagination portion for this particular tap?
e
@justin_fung is it in a public repo?
j
Here you go!
e
Ok, I see your implementation in https://github.com/justincfung/tap-afterpay/blob/246bb35f7d9da15ed0fdda211dc5f864091f59c2/tap_afterpay/client.py#L54-L71. Are there any public API docs on how pagination works?
j
Open in Slack
PreviousNext
Powered by