Hi all. Is there a way to specify a header that wi...
# singer-tap-development
a
Hi all. Is there a way to specify a header that will be sent to all requests? I have to send headers['ClientId'] along with all requests in
client.py
I overridden the http_headers function
Copy code
@property
    def http_headers(self) -> dict:
        """Return the http headers needed."""
        headers = {}
        if "user_agent" in self.config:
            headers["User-Agent"] = self.config.get("user_agent")
        if "client_id" in self.config:
            headers["ClientId"] = self.config.get("client_id")
        # If not using an authenticator, you may also provide inline auth headers:
        # headers["Private-Token"] = self.config.get("auth_token")
        return headers
I expected that on a request to get a token using the OAuthAuthenticator class, headers["ClientId"] would be automatically added to the request, but now this does not happen
r
Docs string on
RESTStream.http_headers
says
If an authenticator is also specified, the authenticator's headers will be combined with
http_headers
when making HTTP requests.
Here is that logic: https://github.com/meltano/sdk/blob/7962ebc9a9a0ff14e77e6aeb1b76d8e5cda2bc73/singer_sdk/streams/rest.py#L274-L301
So your authorization request requires the
ClientId
header?
a
yes
By the way, I just checked in postman and made a request without this header, and still got a token (in the documentation they write that this is a required field) Based on this, it would be logical that the code in the screenshot should also return some kind of response (but I got 401)
message has been deleted
r
Ah, so that changes things a bit then. Can you show the Postman request you are making and your authenticator implementation?
a
curl --location --request POST 'https://sandbox-sa.hostedrmm.com/cwa/api/v1/apitoken' \ --header 'Content-Type: application/json' \ --header 'ClientId: aaaaaaae-db16-4bc9-81c2-947155e531fe' \ --data-raw '{ "username":"xxxxxxx", "password":"xxxxxxx" }'
r
Without the
ClientId
header?
a
with it
but when i run it without, it also work
r
I thought you said it works without it?
a
yes, it can work without
r
Ok, so it's not necessary to provide it.
a
If they write in the documentation that this field must be specified, it is possible that the scope is managed due to this.
r
Right. Can you share your
auth.py
as well?
a
But now I want to make it work somehow. (and then I will add what is necessary)
here is client and auth
r
Confirming in your config,
domain
is
<http://sandbox-sa.hostedrmm.com|sandbox-sa.hostedrmm.com>
?
a
yes
message has been deleted
r
What happens if you make the cURL request without the
Content-Type: application/json
header?
a
It's just a terrible api. I'm getting 401 errors (this seems to be the problem)
r
Yeah, so there doesn't look to be any way to override specifics of the authorization request, apart from to copy the
update_access_token
implementation into your authenticator class and make the changes you need - pretty sure that would just be changing this line to
Copy code
token_response = <http://requests.post|requests.post>(self.auth_endpoint, data=auth_request_payload, headers={"Content-Type": "application/json"})
You could submit an issue to the SDK for a feature that would support configuration of request headers in
update_access_token
. I don't know if this is as simple as just passing
auth_headers
through... @aaronsteers @edgar_ramirez_mondragon
a
It's strange that there is no way to pass headers ... Then what function does it refer to?
If an authenticator is also specified, the authenticator's headers will be combined with http_headers when making HTTP requests.
r
By default if you provide an authenticator,
auth_headers
will be combined with
http_headers
in
RESTStream
request sessions. https://meltano.slack.com/archives/C01PKLU5D1R/p1669287517914089?thread_ts=1669286521.478099&amp;cid=C01PKLU5D1R
a
But you can see that this does not happen in the update_access_token function. Maybe I'm doing something wrong and I have another function that should request the first token?
r
Are you trying to pass the
ClientId
header now?
a
I don't see build_prepared_request being called somewhere before this function (which concatenates headers)
Are you trying to pass the
ClientId
header now?
But where can I put it?
r
No because
build_prepared_request
is specific to
RESTStream
, not an authenticator class.
a
oh, for some reason I thought that the authenticator would also use the REST stream
Is it really possible that I can't do what I need?
r
It is possible, read this again: https://meltano.slack.com/archives/C01PKLU5D1R/p1669290381801209?thread_ts=1669286521.478099&amp;cid=C01PKLU5D1R If you want
ClientId
header as well, change to
Copy code
token_response = <http://requests.post|requests.post>(self.auth_endpoint, data=auth_request_payload, headers={"Content-Type": "application/json", "ClientId", self.config['client_id']})
a
ahh, okay, will try right now
thanks
so, it should be somethink like this? https://gist.github.com/26690fdda5d4b1f73d99883185dd748a
Again error (i can`t make the same error in POSTMAN)
What confuses me is that it encoded the special characters that were in the password (I donโ€™t know if this is normal) Otherwise it looks like it's a valid request
look like i know where is problem I try make POST with raw text, and receive the same error
when i send raw json, in POSTMAN everything is fine
r
a
What would I do without you. You just understand in one word. thanks for the help
r
Copy code
token_response = <http://requests.post|requests.post>(self.auth_endpoint, json=auth_request_payload, headers=headers)
Once you do that, you may not even need the
Content-Type
header.
a
Not only is this my beginning in the world of taps and meltano, but I have never worked with Python either.๐Ÿซ ๐Ÿ˜ช
r
Yeah, there's bit of a learning curve but you seem to have a grasp on the fundamentals. ๐Ÿ™‚
a
I have almost 5 years of node.js experience
r
Hopefully Python isn't putting you off! ๐Ÿ˜…
a
Can you tell me how can I set the default expiration property?
r
message has been deleted
a
ah that's how it works, thanks, I've already started building a huge mess
r
Yeah, maybe it would help if you watched some tutorials on inheritance in Python, since that is a pretty key part of using the SDK.
a
Yeah, you showed me the previous screenshot, and I read a little about it.
And another next error... I got a token, it signed up in Authorization, but when I went further I got an error on the first stream (users)
i think it`s not ok
after this i recieve 401
give a couple of minutes. I think i know reason
It seems to start working and i recieved data. Do you know if this tap will use the received token for all streams? The fact is that it is very difficult to update the token for this product. The token must be renewed before its expiration date expires. I'm not sure if this is possible
r
Yes, it will use that token for the stream the authentication was instantiated on and any others inheriting from it.
update_access_token
should handle getting you a new token if one expires during runtime.
a
usually the token is updated as soon as a 401 error is received. (It means that the life of the token has ended and you need to get a new one). But this product needs to update the token before we get a 401 error. I have an idea to set the default-expiration to be less than 20 minutes, and renew the token if possible. If the update fails, get a new one.
@Reuben (Matatika) could you please review my authenticator? https://gist.github.com/80486df13a81f6add0927c3a4ca36a93
r
https://gist.github.com/Volna13/80486df13a81f6add0927c3a4ca36a93#file-auth-py-L45 You might run into problems here as I'm assuming
self.access_token
is a string (i.e. not JSON). I don't know the API, but you maybe want something like
Copy code
token_response = <http://requests.post|requests.post>(refresh_auth_endpoint, json={"AccessToken": self.access_token}, headers=self.def_headers)
https://gist.github.com/Volna13/80486df13a81f6add0927c3a4ca36a93#file-auth-py-L59 I don't think you actually need the explicit
ClientId
header anymore, as
<http://requests.post|requests.post>
should set it automatically given a value for the
json
kwarg. https://gist.github.com/Volna13/80486df13a81f6add0927c3a4ca36a93#file-auth-py-L79-L84 You don't need to check token validity here with
self.is_token_valid()
as
update_access_token
is only ever called when
self.is_token_valid()
returns
False
anyway. Think this whole bit can change to
Copy code
token = self.refresh_exist_token() if self.access_token else self.get_token()
I think eventually, you could combine the logic of
refresh_exist_token
and
get_token
as they are very similar. Save that for another day though.
a
You might run into problems here as I'm assuming
self.access_token
is a string (i.e. not JSON). I don't know the API, but you maybe want something like
No, no, this is their stupid api again. This is how it looks like in POSTMAN(1 img) (in fact, I already checked this code and it works)
I don't think you actually need the explicit
ClientId
header anymore, as
<http://requests.post|requests.post>
should set it automatically given a value for the
json
kwarg.
ok, i will check, thanks
You don't need to check token validity here with
self.is_token_valid()
as
update_access_token
is only ever called when
self.is_token_valid()
returns
False
anyway. Think this whole bit can change to
Good catch. I will think Thanks for review