https://linen.dev logo
#singer-tap-development
Title
# singer-tap-development
j

jacob_mulligan

05/10/2023, 10:01 AM
I'm developing a tap for a GraphQL API end point, my first time making a Singer tap. I have a question about how to handle pagination with a GraphQL Stream. Here's about what I have now for a stream called
EntriesStream
.
Copy code
class EntriesStream(HealthieStream):
    """Entries stream."""
    name = "entries"
    schema = th.PropertiesList(
        ...
    ).to_dict()

    primary_keys = ["id"]
    replication_key = None
    query = """
        entries (
            is_org: true
        ) {
            added_by_user_id
            category
            created_at
            emotions
            emotions_string
            entries_in_average
            external_id
            external_id_type
            has_subentries
            healthiness_info_hex_value
            hide_from_main_feed
            id
            metric_stat_string
            name
        }
        """
I can specify an
offset
and
page_size
like this:
Copy code
query = """
        entries (
            is_org: true
            page_size: {page_size}
            offset: {offset}
        ) {...}
But I'm not sure how to actually set this up to make multiple API calls with different offsets.. I see the docs for BaseOffsetPaginator, but also see for the GraphQLStream docs that I can overwrite the prepare_request_payload method and set next_page_token. How should I set up this Stream to make multiple API calls until all data has been fetched? Thank you! 🫂