tap jira the API is utilizing paging and the page size is ha Meltano #singer-taps

tap-jira - the API is utilizing paging and the pag...

jan_soubusta

01/25/2024, 2:30 PM

tap-jira - the API is utilizing paging and the page size is hardcoded to 10: https://github.com/MeltanoLabs/tap-jira/blob/main/tap_jira/streams.py#L388 I suggest to extract it to the config. Any objections? Anyone here who would merge my PR?

Edgar Ramírez (Arch.dev)

01/25/2024, 6:15 PM

Yup, PRs welcome.

jan_soubusta

01/26/2024, 7:32 AM

Well, I realized that I don't know how to do it 😉

path

is class variable. In such a scope I don't know how to access self.config. In a class method having

self

as first argument it's clear. But here, I don't know. Would you accept PR changing just the static value 10 to 100? Based on Jira REST API DOC they recommend to set this value large a let server to decide what is the paging threshold...

Edgar Ramírez (Arch.dev)

01/26/2024, 7:16 PM

path

is a class attribute but it's always accessed as an instance attribute and never accessed at the class level, e.g. in class methods or as

<Class>.path

. But you don't even need that. It's probably better to remove the url param from the path and update

get_url_params

to add

maxResults

to the params dictionary.

jan_soubusta

01/26/2024, 7:20 PM

Gotcha. WIP

🙌 1

jan_soubusta

01/29/2024, 12:20 PM

Please, review and merge: https://github.com/MeltanoLabs/tap-jira/pull/64

👀 1

jan_soubusta

01/31/2024, 8:29 AM

Thanks for merge. Now it works for me with both Snowflake and MotherDuck as targets. But I found out that it is running for a long time even when I re-execute it with existing state. I reviewed the code and I see that config.start_date is always applied as filter, but the stored state is not utilized. So when I extract/load all Jira tickets updated starting from let's say 1.1.2023, it is running for tens of minutes. Initial load, it's OK. But I expect that when I execute it second time, the start_date in Jira filter will be filled from the stored state, e.g. by 31.1.2024, and it will be running very fast. Any advice what am I doing wrong or what could I fix to make incremental loads fast?

jan_soubusta

01/31/2024, 8:30 AM

Btw exactly this works well with tap-github without a need to do anything special, it works out of the box

jan_soubusta

01/31/2024, 8:45 AM

I just checked both. Isn't the issue caused by wrongly set replication key? tap-jira:

Copy code

replication_key = "id"
    replication_method = "INCREMENTAL"

tap-github (commits):

Copy code

replication_key = "commit_timestamp"

Btw. replication_method is not specified in tap-github.

Open in Slack

Previous Next