https://linen.dev logo
#singer-tap-development
Title
# singer-tap-development
d

Denis I.

02/22/2023, 11:06 PM
Hi, I’m building a tap for a custom API and it has tricky paginated responses:
Copy code
{
  "objects": [
    {
      "id": 1,
      "child_id": 1,
      "type_id": 1
    },
    {
      "id": 2,
      "child_id": 2,
      "type_id": 2
    },
    {
      "id": 3,
      "child_id": 3,
      "type_id": 2
    }
  ],
  "types": [
    {
      "id": 1,
      "name": "1"
    },
    {
      "id": 2,
      "name": "2"
    }
  ],
  "hasMore": true
}
Copy code
{
  "objects": [
    {
      "id": 4,
      "child_id": 4,
      "type_id": 1
    },
    {
      "id": 5,
      "child_id": 5,
      "type_id": 1
    }
  ],
  "types": [
    {
      "id": 1,
      "name": "1"
    }
  ],
  "hasMore": false
}
So, I made it work with a RESTStream (
records_jsonpath = "$.objects[*]"
) which passes it’s
child_id
to child RESTStream. It gives me two fancy outputs:
Copy code
objects:
{"id": 1, "child_id": 1, "type_id": 1}
{"id": 2, "child_id": 2, "type_id": 2}
{"id": 3, "child_id": 3, "type_id": 2}
{"id": 4, "child_id": 4, "type_id": 1}
{"id": 5, "child_id": 5, "type_id": 1}
Copy code
children:
{"id": 1, "parent_id": 1, "type_id": 1}
{"id": 2, "parent_id": 2, "type_id": 2}
{"id": 3, "parent_id": 3, "type_id": 2}
{"id": 4, "parent_id": 4, "type_id": 1}
{"id": 5, "parent_id": 5, "type_id": 1}
Everything works fine at this step with a pagination token linked to the
hasMore
value. The tricky part here is the
types
list. Each response page with
objects
and
types
could have same
types
on each page, but I want to have
types
as a separate stream with only unique elements appeared on all of the pages:
Copy code
types:
{"id": 1, "name": "1"}
{"id": 2, "name": "2"}
Is there a proper way to solve it?