steve_clarke
06/22/2023, 8:54 AMauth=
the request works and I receive my desired results as the appropriate AWS credentials were added to the request.
https://github.com/s7clarke10/tap-rest-api-msdk/blob/06105b1c1ed45dc98ae35940acbab3e14c21293f/tap_rest_api_msdk/tap.py#L554
r = requests.get(self.config["api_url"] + path, auth=self.aws_auth, params=params, headers=headers)
However when I attempt to set this authenticator it does appear to set any AWS credential in the request as I would like.
https://github.com/s7clarke10/tap-rest-api-msdk/blob/06105b1c1ed45dc98ae35940acbab3e14c21293f/tap_rest_api_msdk/tap.py#L655
return AWS4Auth(
self.aws_connection.credentials.access_key,
self.aws_connection.credentials.secret_key,
self.aws_connection.region,
self.aws_connection.aws_service,
aws_session=self.aws_connection.credentials.token
)
The SDK documentation suggested I should be able to use other authenticators like aws4auth. Some guidance would be so helpful, I have been really banging my head against the wall on this.
Note: All the other extensions I have added to this tap like supporting multiple forms of authentication and paginators do work really well. Just needing to get AWS Authentication working as well via this method using AWS4AUTH as I require signed credential.
Thanks 😀steve_clarke
06/23/2023, 5:33 AMprepare_request
in my streams.py and override it to pass the auth
parameter when creating a PreparedRequest.
# TODO: This SDK function has been included and overloaded because we need to
# pass in auth to the request. Would like this as an enhancement to the SDK.
#
# Would like to remove this function to have a vanilla implementation.
def prepare_request(
self,
context: Optional[dict], # Replaced to remove Type Error
next_page_token: Optional[Any] # Replaced to remove Type Error
# context: dict | None,
# next_page_token: _TToken | None,
) -> requests.PreparedRequest:
"""Prepare a request object for this stream.
If partitioning is supported, the `context` object will contain the partition
definitions. Pagination information can be parsed from `next_page_token` if
`next_page_token` is not None.
Args:
context: Stream partition or context dictionary.
next_page_token: Token, page number or any request argument to request the
next page of data.
Returns:
Build a request with the stream's URL, path, query parameters,
HTTP headers and authenticator.
"""
http_method = self.rest_method
url: str = self.get_url(context)
params: dict | str = self.get_url_params(context, next_page_token)
request_data = self.prepare_request_payload(context, next_page_token)
headers = self.http_headers
auth = self.http_auth # Extended Meltano SDK with this line
return self.build_prepared_request(
method=http_method,
url=url,
auth=auth, # Extended Meltano SDK with this line
params=params,
headers=headers,
json=request_data,
)
Would love to discuss an extension to the SDK to support the auth parameter so you can easily use supported requests based authenticators like AWS4AUTH. It would seem there would need to be an additional variable associated with the Authenticators to set the auth
, by default the auth would be None. It would be nice if you could optionally provide the auth in a supported manner when you create an authenticator.
@edgar_ramirez_mondragon 👀 - I know you would have a view on this (some advice would be very much appreciated). Thankssteve_clarke
06/26/2023, 11:17 AMedgar_ramirez_mondragon
07/03/2023, 3:36 PMv0.20.0
) and it got updated after the refactor? That’s happened to me before 😅. I’m glad you figured it out.