Hi all. Is there a way to specify a header that will be sent to all requests? I have to send headers['ClientId'] along with all requests in

client.py

I overridden the http_headers function

@property
    def http_headers(self) -> dict:
        """Return the http headers needed."""
        headers = {}
        if "user_agent" in self.config:
            headers["User-Agent"] = self.config.get("user_agent")
        if "client_id" in self.config:
            headers["ClientId"] = self.config.get("client_id")
        # If not using an authenticator, you may also provide inline auth headers:
        # headers["Private-Token"] = self.config.get("auth_token")
        return headers

I expected that on a request to get a token using the OAuthAuthenticator class, headers["ClientId"] would be automatically added to the request, but now this does not happen

Docs string on

RESTStream.http_headers

says

If an authenticator is also specified, the authenticator's headers will be combined with

http_headers

when making HTTP requests.

Here is that logic: https://github.com/meltano/sdk/blob/7962ebc9a9a0ff14e77e6aeb1b76d8e5cda2bc73/singer_sdk/streams/rest.py#L274-L301

yes

Ah, so that changes things a bit then. Can you show the Postman request you are making and your authenticator implementation?

curl --location --request POST 'https://sandbox-sa.hostedrmm.com/cwa/api/v1/apitoken' \ --header 'Content-Type: application/json' \ --header 'ClientId: aaaaaaae-db16-4bc9-81c2-947155e531fe' \ --data-raw '{ "username":"xxxxxxx", "password":"xxxxxxx" }'

Without the

ClientId

header?

with it

I thought you said it works without it?

yes, it can work without

Ok, so it's not necessary to provide it.

If they write in the documentation that this field must be specified, it is possible that the scope is managed due to this.

Right. Can you share your

auth.py

as well?

But now I want to make it work somehow. (and then I will add what is necessary)

Confirming in your config,

domain

is

<http://sandbox-sa.hostedrmm.com|sandbox-sa.hostedrmm.com>

?

yes

What happens if you make the cURL request without the

Content-Type: application/json

header?

It's just a terrible api. I'm getting 401 errors (this seems to be the problem)

Yeah, so there doesn't look to be any way to override specifics of the authorization request, apart from to copy the

update_access_token

implementation into your authenticator class and make the changes you need - pretty sure that would just be changing this line to

token_response = <http://requests.post|requests.post>(self.auth_endpoint, data=auth_request_payload, headers={"Content-Type": "application/json"})

You could submit an issue to the SDK for a feature that would support configuration of request headers in

update_access_token

. I don't know if this is as simple as just passing

auth_headers

through... @aaronsteers @edgar_ramirez_mondragon

It's strange that there is no way to pass headers ... Then what function does it refer to?

If an authenticator is also specified, the authenticator's headers will be combined with http_headers when making HTTP requests.

By default if you provide an authenticator,

auth_headers

will be combined with

http_headers

in

RESTStream

request sessions. https://meltano.slack.com/archives/C01PKLU5D1R/p1669287517914089?thread_ts=1669286521.478099&cid=C01PKLU5D1R

But you can see that this does not happen in the update_access_token function. Maybe I'm doing something wrong and I have another function that should request the first token?

Are you trying to pass the

ClientId

header now?

I don't see build_prepared_request being called somewhere before this function (which concatenates headers)

No because

build_prepared_request

is specific to

RESTStream

, not an authenticator class.

oh, for some reason I thought that the authenticator would also use the REST stream

It is possible, read this again: https://meltano.slack.com/archives/C01PKLU5D1R/p1669290381801209?thread_ts=1669286521.478099&cid=C01PKLU5D1R If you want

ClientId

header as well, change to

token_response = <http://requests.post|requests.post>(self.auth_endpoint, data=auth_request_payload, headers={"Content-Type": "application/json", "ClientId", self.config['client_id']})

ahh, okay, will try right now

What would I do without you. You just understand in one word. thanks for the help

token_response = <http://requests.post|requests.post>(self.auth_endpoint, json=auth_request_payload, headers=headers)

Once you do that, you may not even need the

Content-Type

header.

Not only is this my beginning in the world of taps and meltano, but I have never worked with Python either.🫠😪

Yeah, there's bit of a learning curve but you seem to have a grasp on the fundamentals. 🙂

I have almost 5 years of node.js experience

Hopefully Python isn't putting you off! 😅

Can you tell me how can I set the default expiration property?

message has been deleted

ah that's how it works, thanks, I've already started building a huge mess

Yeah, maybe it would help if you watched some tutorials on inheritance in Python, since that is a pretty key part of using the SDK.

Yeah, you showed me the previous screenshot, and I read a little about it.

Yes, it will use that token for the stream the authentication was instantiated on and any others inheriting from it.

update_access_token

should handle getting you a new token if one expires during runtime.

usually the token is updated as soon as a 401 error is received. (It means that the life of the token has ended and you need to get a new one). But this product needs to update the token before we get a 401 error. I have an idea to set the default-expiration to be less than 20 minutes, and renew the token if possible. If the update fails, get a new one.

https://gist.github.com/Volna13/80486df13a81f6add0927c3a4ca36a93#file-auth-py-L45 You might run into problems here as I'm assuming

self.access_token

is a string (i.e. not JSON). I don't know the API, but you maybe want something like

token_response = <http://requests.post|requests.post>(refresh_auth_endpoint, json={"AccessToken": self.access_token}, headers=self.def_headers)

https://gist.github.com/Volna13/80486df13a81f6add0927c3a4ca36a93#file-auth-py-L59 I don't think you actually need the explicit

ClientId

header anymore, as

<http://requests.post|requests.post>

should set it automatically given a value for the

json

kwarg. https://gist.github.com/Volna13/80486df13a81f6add0927c3a4ca36a93#file-auth-py-L79-L84 You don't need to check token validity here with

self.is_token_valid()

as

update_access_token

is only ever called when

self.is_token_valid()

returns

False

anyway. Think this whole bit can change to

token = self.refresh_exist_token() if self.access_token else self.get_token()

You might run into problems here as I'm assuming

self.access_token

is a string (i.e. not JSON). I don't know the API, but you maybe want something like

No, no, this is their stupid api again. This is how it looks like in POSTMAN(1 img) (in fact, I already checked this code and it works)

I don't think you actually need the explicit

ClientId

header anymore, as

<http://requests.post|requests.post>

should set it automatically given a value for the

json

kwarg.

ok, i will check, thanks

You don't need to check token validity here with

self.is_token_valid()

as

update_access_token

is only ever called when

self.is_token_valid()

returns

False

anyway. Think this whole bit can change to

Good catch. I will think Thanks for review