Hi, I’m building a tap for a custom API and it has tricky paginated responses:

{
  "objects": [
    {
      "id": 1,
      "child_id": 1,
      "type_id": 1
    },
    {
      "id": 2,
      "child_id": 2,
      "type_id": 2
    },
    {
      "id": 3,
      "child_id": 3,
      "type_id": 2
    }
  ],
  "types": [
    {
      "id": 1,
      "name": "1"
    },
    {
      "id": 2,
      "name": "2"
    }
  ],
  "hasMore": true
}

{
  "objects": [
    {
      "id": 4,
      "child_id": 4,
      "type_id": 1
    },
    {
      "id": 5,
      "child_id": 5,
      "type_id": 1
    }
  ],
  "types": [
    {
      "id": 1,
      "name": "1"
    }
  ],
  "hasMore": false
}

So, I made it work with a RESTStream (

records_jsonpath = "$.objects[*]"

) which passes it’s

child_id

to child RESTStream. It gives me two fancy outputs:

objects:
{"id": 1, "child_id": 1, "type_id": 1}
{"id": 2, "child_id": 2, "type_id": 2}
{"id": 3, "child_id": 3, "type_id": 2}
{"id": 4, "child_id": 4, "type_id": 1}
{"id": 5, "child_id": 5, "type_id": 1}

children:
{"id": 1, "parent_id": 1, "type_id": 1}
{"id": 2, "parent_id": 2, "type_id": 2}
{"id": 3, "parent_id": 3, "type_id": 2}
{"id": 4, "parent_id": 4, "type_id": 1}
{"id": 5, "parent_id": 5, "type_id": 1}

Everything works fine at this step with a pagination token linked to the

hasMore

value. The tricky part here is the

types

list. Each response page with

objects

and

types

could have same

types

on each page, but I want to have

types

as a separate stream with only unique elements appeared on all of the pages:

types:
{"id": 1, "name": "1"}
{"id": 2, "name": "2"}

Is there a proper way to solve it?