https://linen.dev logo
#plugins-general
Title
# plugins-general
d

douwe_maan

12/18/2020, 4:34 PM
@jason_waldrip Since you're primarily dealing with jobs, be aware of the limitation described here, though: https://gitlab.com/meltano/tap-gitlab/-/issues/31. The tap wants to make sure that when a job is still running or in some other non-terminal state during one run, its transition to a terminal states (success, failure) in a next state is picked up, so that the record is output again by the tap for the target to upsert (or append) at the destination. Since the Jobs API doesn't allow us to filter by "updated_at" or "state_changed_at", the way this is currently implemented is by always going back to the oldest job that was still in a non-terminal state, and then we have to iterate over every job since that point. This means that if you have some old job in your batch of 13000 pages that for whatever reason (possibly a GitLab bug) is still in a non-terminal state, the tap will always be going back unnecessarily far if that job will never transition out of its state. As a workaround, you can move up the start_date to a "safe" date after which all jobs can be expected to transition normally. So after your initial massive run, consider setting the
start_date
to yesterday's date explicitly. That will then override the older date stored in the state payload.