Is there a best practice for handling api’s that return one record per api call? I was thinking of treating each record as it's own “page”, then paginating through all the records.

Hi @collin_prather! AFAIK no best practice has been established so far. There are APIs that return a minimal object in the bulk endpoint and more details in the individual one. I think people have been using child streams to handle those cases, but it definitely depends on where the individual IDs for each record are coming from. Does the API call look something like

/api/MyEntity/<id>

?

Ah, I see! Yes, unfortunately there is no bulk API available. And yes, that is exactly what the API path looks like!

Yeah, this one is tough to handle. I have done a couple of things, both creating a "null parent" stream and calling children from it (e.g. in tap-immuta https://github.com/immuta/tap-immuta/blob/4b0789ba879a5f0d4120bb44011b857f67b83ff8/tap_immuta/streams.py#L54) and just passing in records for specific entities to sync (e.g. tap-hightouch) https://github.com/immuta/tap-hightouch/blob/050fb9eace4d3558f23e6c2804b93bc8d9a97169/tap_hightouch/streams.py#L18

Great examples! Super helpful, I'm going to take a look. I've been tinkering with the hacker news api. Your approach in tap-hightouch looks like it’d be a straightforward solution! Now I need to do some reading about partitions 🤓