Hi all, We have requirement to use tap-gitlab and extract pipelines and jobs and we running the tap with list of groups.Due to very huge data the tap-gitlab is running more than 16hours to extract the data and at the end getting timeout issue. Is there any setting/resolution so that we can use to handle this situation?

Hi @Haruno izumi! What variant of tap-gitlab is this? A look at your

meltano.yml

might also help 🙂

We are using meltano default with latest release code.This is .yml configurations we are using

Thanks. So, even if the pipeline is interrupted by an error at any point, Meltano should be able to record the tap's state up to that point and start the next run from that checkpoint. Did you use

meltano run

?

Does that mean,if Suppose it extracted only few pipelines and jobs details for a specific project and then after it failed.will the state be updated till where it extracted data?

Correct

We are running the tap in k8s and due to long running hours pods terminating and state wasn't getting updated

Oh, ok so it's an ephemeral environment. Then you might wanna configure an external state backend.

We have State backend configured to AWS

Ok. So if you used

meltano run

, you should be able to inspect the state using

meltano state list

and

meltano state get

.

Does that mean...even it fails/terminates as it updating the state everytime we can just re trigger the tap again?

Right

Got it! One more query, I have seen the release v2.1.0 and v2.0.0 have different code for gitlab tap.Would you be able to tell..what is the difference and which is more preferable? Because when I tried to use the v2.0.0 release upon meltano install it failed with origin/master not found error while checking out

Ah I see! The v2.1.0 tag was created from the "legacy" version of the tap but v2.0.0 was created from the "newer" SDK-based version. I've honestly not been too involved with the tap, so I'm not sure how the switch happened, but I'd probably prefer pointing to

@main

in my

pip_url

.