Following up on @visch’s post: I’m in the process of building a ClickUp tap, and his first item is where I’m stuck right now. What’s the best practice for handling APIs that don’t return pagination hints for a given endpoint? ClickUp’s Get Tasks operation doesn’t provide a record count total, nor does it provide pagination hints/props. It uses a

page

0-based index for requests, and returns a maximum of 100 items. Unfortunately, if there are less than 100 items, and you pass a

page

querystring param that should return 0 rows, it merrily returns the complete set of records. For example, this request (using ClickUp’s Apiary Mock endpoint, so you can run it as-is) returns a list of valid workspaces for the ClickUp Team with a

team_id

of `512`:

curl <https://private-anon-1ab3e1ce0e-clickup20.apiary-mock.com/api/v2/team/512/space>

Note that there are only two workspaces in the response. With a maximum of 100 records per-page, adding a

page

querystring param should return zero records. However, If I add a

page

querystring param (per the API docs) to return the “next” page of records…

curl <https://private-anon-1ab3e1ce0e-clickup20.apiary-mock.com/api/v2/team/512/space?page=1>

I get back the same two entries. This is true for live API requests, also. Is there a recommended approach for dealing with APIs like this? That is, other than submitting a request to ClickUp to fix their API, which I have already done. 😄

@dustin_miller - Are there any hints at all that either (1) we're already on the last page or (2) that the returned "page 2" isn't really so?

That’s…FUN 😄 I was just wondering if storing the

id

of the first record returned would work, but that depends on the sort order being consistent from one request to another (not all endpoints support a

sort_by

param)

I don't know the internals of

id

to confirm/question the approach, but yes, you can count the records and stop when n<100 - and also any heuristic to keep from looping when the last page has 100 items.

But that’s only because I’m unsure ……… What’s the default behavior if an ostensibly unique

primary_keys

result is repeated within a given stream? Last one wins?

In case it's not clear, you can put any non-falsy thing you want inside

next_page_token

- and you'll have the prior token when you're evaluating the next one. So, for this use case, you'll probably want to make it a dict so you can keep more detailed info from one page to the next.

oh that’s a good point, I don’t have to pass that token into the request, after all.

Yeah - the test of whether continue is

if next_page_token

- so as long as its not empty or None or similar, anything else will keep the flow going - and then you can smartly compare whatever vars are needed.

🛎️

What’s the default behavior if an ostensibly unique 

primary_keys

 result is repeated within a given stream? Last one wins?

Yes - but I'd be slightly worried about deduping before the merge. On some targets they create a batch and merge upsert and/or insert the result to the target table. Many dedupe also before the merge upsert, but that could vary. ("I'm not sure" is the shorter answer.)

Yes - but I’d be slightly worried about deduping before the merge. [snip]

Fair point. If I want this to be usable for any

target-

I’d need to make sure I handle that myself in the tap. Would that best be wrapped into a

stream_map

or would it make sense to put some basic “last one wins” logic (or whatever seems correct) into

post_process

in

client.py

?

If you get pagination workarounds of some sort, hopefully you won't end up sending duplicate records in the same sync operation. That said, if all else fails, you could keep a hash (or other tracker) of each record you've seen and in

post_process()

you can simply return

None

to skip a record entirely if it has already been sent.

@dustin_miller Alredy made a clickup tap https://github.com/AutoIDM/tap-clickup feel free to give it a spin! We're getting close to publishing it to MeltanoHub / Stitch

Sweet, thanks @visch!