Hello, I'm new to meltano and singer taps and I'm ...
# troubleshooting
n
Hello, I'm new to meltano and singer taps and I'm currently building my first REST tap. the API I'm working with supports both filtering and sorting, but the only relevant field I can use is a GUID and I have some issues with creating the correct params in get_url_params. I managed to get it to work with a bare-bones requests request in python, where the following params create this url:
Copy code
r = younium.get_data(
    endpoint='Subscriptions',
    params={
        'orderBy': 'id desc',
        'Filter': "id gt guid'e0a6a6d4-9407-40c5-d041-08db03ef442c'"})

r.request.url -> '<https://api.younium.com/Subscriptions?orderBy=id+desc&Filter=id+gt+guid%27e0a6a6d4-9407-40c5-d041-08db03ef442c%27>'
my get_url_params looks like this:
Copy code
def get_url_params(
        self, context: Optional[dict], next_page_token: Optional[Any]
    ) -> Dict[str, Any]:
        """Return a dictionary of values to be used in URL parameterization."""
        params: dict = {"orderBy": "id asc"}

        replication_key_value = self.get_starting_replication_key_value(context)

        if next_page_token:
            params["PageNumber"] = next_page_token
        if replication_key_value:
            params["Filter"] = f"id gt guid'{replication_key_value}'"
        # print(params)
        return params
it works when I run a full refresh but when I try "normal" refresh, it fails with this error:
Bad Request for path: /Subscriptions
. my guess is that
params["Filter"] = f"id gt guid'{replication_key_value}'"
for some reason isn't parsed correctly, any advice how I can inspect the request url?
ok, the issue was actually that the API sends back a 400 response if there's no id that's greater than the last id. is there a recommended way to get around this?
e
Hi @niclas_roos, so the 400 response in this case means you're in the last page of results?
n
well, right now there's no results at all, because no new records have been created so I just get the 400
my plan is to add a post_process to the stream, like so:
Copy code
def post_process(self, row: dict, context: Optional[dict]) -> dict:
        """As needed, append or transform raw data to match expected structure."""
        
        replication_key_value = self.get_starting_replication_key_value(context)

        if replication_key_value != row['id']:
            row['legalEntity'] = self.config.get('legal_entity')
        else:
            row = None

        return row
and change the filter to >= instead of >
that seems to have done the trick!