Hi, I been trying to use a custom authenticator as...
# singer-tap-development
s
Hi, I been trying to use a custom authenticator as per the documentation here request-aws4auth . I am stuck with how I get the resultant AWS4Auth object adding in the required requests auth= parameter. The tap is an enhancement to tap-rest-api-msdk where I am adding support to access a AWS end-point. My feature branch is located here: https://github.com/s7clarke10/tap-rest-api-msdk/tree/feature/add_aws_authentication. When I run this block of code with a basic request passing in the
auth=
the request works and I receive my desired results as the appropriate AWS credentials were added to the request. https://github.com/s7clarke10/tap-rest-api-msdk/blob/06105b1c1ed45dc98ae35940acbab3e14c21293f/tap_rest_api_msdk/tap.py#L554
Copy code
r = requests.get(self.config["api_url"] + path, auth=self.aws_auth, params=params, headers=headers)
However when I attempt to set this authenticator it does appear to set any AWS credential in the request as I would like. https://github.com/s7clarke10/tap-rest-api-msdk/blob/06105b1c1ed45dc98ae35940acbab3e14c21293f/tap_rest_api_msdk/tap.py#L655
Copy code
return AWS4Auth(
                self.aws_connection.credentials.access_key,
                self.aws_connection.credentials.secret_key,
                self.aws_connection.region,
                self.aws_connection.aws_service,
                aws_session=self.aws_connection.credentials.token
            )
The SDK documentation suggested I should be able to use other authenticators like aws4auth. Some guidance would be so helpful, I have been really banging my head against the wall on this. Note: All the other extensions I have added to this tap like supporting multiple forms of authentication and paginators do work really well. Just needing to get AWS Authentication working as well via this method using AWS4AUTH as I require signed credential. Thanks 😀
Okay, I have a working patch for this I need to include the
prepare_request
in my streams.py and override it to pass the
auth
parameter when creating a PreparedRequest.
Copy code
# TODO: This SDK function has been included and overloaded because we need to
    # pass in auth to the request. Would like this as an enhancement to the SDK.
    #
    # Would like to remove this function to have a vanilla implementation.
    def prepare_request(
        self,
        context: Optional[dict], # Replaced to remove Type Error
        next_page_token: Optional[Any] # Replaced to remove Type Error
#        context: dict | None,
#        next_page_token: _TToken | None,
    ) -> requests.PreparedRequest:
        """Prepare a request object for this stream.

        If partitioning is supported, the `context` object will contain the partition
        definitions. Pagination information can be parsed from `next_page_token` if
        `next_page_token` is not None.

        Args:
            context: Stream partition or context dictionary.
            next_page_token: Token, page number or any request argument to request the
                next page of data.

        Returns:
            Build a request with the stream's URL, path, query parameters,
            HTTP headers and authenticator.
        """
        http_method = self.rest_method
        url: str = self.get_url(context)
        params: dict | str = self.get_url_params(context, next_page_token)
        request_data = self.prepare_request_payload(context, next_page_token)
        headers = self.http_headers
        auth = self.http_auth  # Extended Meltano SDK with this line

        return self.build_prepared_request(
            method=http_method,
            url=url,
            auth=auth, # Extended Meltano SDK with this line
            params=params,
            headers=headers,
            json=request_data,
        )
Would love to discuss an extension to the SDK to support the auth parameter so you can easily use supported requests based authenticators like AWS4AUTH. It would seem there would need to be an additional variable associated with the Authenticators to set the
auth
, by default the auth would be None. It would be nice if you could optionally provide the auth in a supported manner when you create an authenticator. @edgar_ramirez_mondragon 👀 - I know you would have a view on this (some advice would be very much appreciated). Thanks
Okay, after a major refactor to clean up code and place classes and functions in logical places it seems you can simply have a custom authenticator return a requests compatible authenticator like AWS4AUTH and it will set the requests.auth for you. I am not totally sure why it wasn't initially working for me, but gave me a real good excuse to clean up additions the tap. My working solution is in this branch https://github.com/s7clarke10/tap-rest-api-msdk/tree/feature/add_aws_authentication
e
Hey @steve_clarke! You probably still had an older version of the SDK installed (i.e. <
v0.20.0
) and it got updated after the refactor? That’s happened to me before 😅. I’m glad you figured it out.