Hi I m facing an error with a child stream in `tap jira` Whi Meltano #troubleshooting

Hi I'm facing an error with a child stream in `tap...

prakhar_srivastava

09/19/2023, 1:31 PM

Hi I'm facing an error with a child stream in

tap-jira

. While the parent stream gets the unique records, the child stream is fetching duplicate records. How can I tackle this?

Reuben (Matatika)

09/19/2023, 1:46 PM

Can you share some logs or the code?

prakhar_srivastava

09/20/2023, 8:46 AM

here you can see, from url, that the record is duplicated

Reuben (Matatika)

09/20/2023, 8:56 AM

You might have to filter duplicates out in

parse_response

if the parent or child stream is erroneously returning multiple records of the same ID - although you might want to consider why this is happening (e.g. is the API response wrong, is a comment with the same ID as another an edit of the same comment).

prakhar_srivastava

09/20/2023, 9:02 AM

thanks for the reply! The parent stream is working alright, with non duplicated records. The api endpoint is also giving the right response, but the child stream is duplicating records

Reuben (Matatika)

09/20/2023, 9:02 AM

Can you share your child stream class then?

prakhar_srivastava

09/20/2023, 9:04 AM

Copy code

class IssueComments(JiraStream):

    """
    <https://developer.atlassian.com/cloud/jira/platform/rest/v3/api-group-issue-comments/#api-rest-api-3-issue-issueidorkey-comment-get>
    """

    """
    name: stream name
    path: path which will be added to api url in client.py
    schema: instream schema
    primary_keys = primary keys for the table
    replication_key = datetime keys for replication
    records_jsonpath = json response body
    """

    name = "issue_comments"

    parent_stream_type = IssueStream

    ignore_parent_replication_keys = True

    path = "/issue/{issue_id}/comment"

    primary_keys = ["id"]

    records_jsonpath = "$[comments][*]"

    instance_name = "comments"

    schema = PropertiesList(
        Property("id", StringType),
        Property("issueId", StringType),
        Property("self", StringType),
        Property(
            "author",
            ObjectType(
                Property("accountId", StringType),
                Property("self", StringType),
                Property("displayName", StringType),
                Property("active", BooleanType),
            ),
        ),
        Property("created", DateTimeType),
        Property("updated", DateTimeType),
        Property(
            "body",
            ObjectType(
                Property("type", StringType),
                Property("version", IntegerType),
                Property(
                    "content",
                    ArrayType(
                        ObjectType(
                            Property("type", StringType),
                            Property(
                                "content",
                                ArrayType(
                                    ObjectType(
                                        Property("type", StringType),
                                        Property("text", StringType),
                                    )
                                ),
                            ),
                        )
                    ),
                ),
            ),
        ),
        Property(
            "updateAuthor",
            ObjectType(
                Property("accountId", StringType),
                Property("self", StringType),
                Property("displayName", StringType),
                Property("active", BooleanType),
            ),
        ),
    ).to_dict()

    def post_process(self, row:dict, context:dict) -> dict:
        row["issueId"] = context["issue_id"]
        return row

prakhar_srivastava

09/20/2023, 9:10 AM

I also observed now that 1st comment is always once 2nd comment is duplicated twice 3rd comment is duplicated thrice.. so on I think something is wrong in the child stream itself but don't have clarity around singer so much

prakhar_srivastava

09/20/2023, 9:13 AM

here you can see the behaviour

prakhar_srivastava

09/20/2023, 9:14 AM

640851 -- occurs once (1st comment) 641018 -- occurs twice (2nd comment) 641099 -- occurs thrice (3rd comment)

Reuben (Matatika)

09/20/2023, 9:19 AM

Can I see the

JiraStream

implementation too?

prakhar_srivastava

09/20/2023, 9:25 AM

https://github.com/prakharcode/tap-jira

prakhar_srivastava

09/20/2023, 9:26 AM

the same behaviour is across all the child streams thinkspin

Reuben (Matatika)

09/20/2023, 9:34 AM

I wonder if it's something to do with this: https://github.com/prakharcode/tap-jira/blob/0eaac5921dac8ef2efbf300305f0cc29b8dc95ae/tap_jira/client.py#L89-L90 Perhaps the

get_next_page_token

logic is causing the increasing duplication. I would have a look into that in more detail.

prakhar_srivastava

09/20/2023, 9:34 AM

I'm also exploring that essentially

prakhar_srivastava

09/20/2023, 9:34 AM

will update but thanks for the time, much appreciated

Reuben (Matatika)

09/20/2023, 9:41 AM

No problem! Good luck! 😁

prakhar_srivastava

09/20/2023, 9:50 AM

solved it

prakhar_srivastava

09/20/2023, 9:52 AM

indeed the problem was pagination logic -- by default the return per page is 100 records and pagination logic is per record based so it was making 1 req for one object which fetched 100 records then another request which fetched the same 100 records over and over till 100 records

prakhar_srivastava

09/20/2023, 9:53 AM

thanks for the direction :)

Reuben (Matatika)

09/20/2023, 10:11 AM

Nice! Glad you figured it out. 😄

3 Views

Open in Slack

Previous Next