Hello tap dev gurus 🙂 We've working on a tap for an API where the endpoints do not like

:

characters to be encoded as

%3A

in parameter strings. For example

curl -H "User-Agent: test" "<https://api.service.cqc.org.uk/public/v1/changes/location?startTimestamp=2024-03-04T00:00:00Z&endTimestamp=2024-03-04T15:32:24Z>"

is ok 🙂 but

curl -H "User-Agent: test" "<https://api.service.cqc.org.uk/public/v1/changes/location?startTimestamp=2024-03-04T00%3A00%3A00Z&endTimestamp=2024-03-04T15%3A32%3A24Z>"

is not 😞 . The version of the SDK we're currently using (which is at least a couple of years old) encodes timestamps in the latter format. To get around this we've got a hack... which is to override the rest stream class's

prepare_request()

method like this...

def prepare_request(
            self, context: Optional[dict], next_page_token: Optional[Any]
        ) -> requests.PreparedRequest:

            request = super().prepare_request(context, next_page_token)
            request.headers["User-Agent"] = self.config["partner_code"]
            request.url = request.url.replace("%3A", ":")
            return request

But is there a more elegant way to control how the SDK encodes special characters? cc @miguel_sotomayor

If you take a look at the docs on get_url_params it explains that you can return a string rather than a dict if you don’t want it to automatically get urlencoded

+1 to Peter's suggestion, that was the express intention of https://github.com/meltano/sdk/pull/1623. @benw-at-birdie do let me know if you have thought on how to make https://sdk.meltano.com/en/v0.36.0/classes/singer_sdk.RESTStream.html#singer_sdk.RESTStream.get_url_params more clear and discoverable.

Thanks both. A much better solution. > let me know if you have thought on how to make https://sdk.meltano.com/en/v0.36.0/classes/singer_sdk.RESTStream.html#singer_sdk.RESTStream.get_url_params more clear and discoverable. I think it's clear and well documented but we just missed it - I wouldn't read too much into it.