Hi community I ve a question about states after a job failed

Question

Hi community I ve a question about states after a job failed using Adswerve target bigquery After a tap fails the next run uses the START DATE set in the config variable instead of picking up the state that failed I saw in the documentation in the end of the page that some loaders might interrupt the pipeline before meltano writes the state Does this happen with the

ripe-musician-59933 · Answer

< curved lunch 43500> Reading through it appears to always emit state whenever a STATE message comes in from the tap and whenever it s done processing a stream

ripe-musician-59933 · Answer

Also yield init state so even if there is an exception right after we get proper state emitted <https github com adswerve target bigquery blob master target bigquery process py L22>

ripe-musician-59933 · Answer

But there may not be any initial state if the target fails pretty much immediately

ripe-musician-59933 · Answer

If the failed pipeline run didn t emit any state Meltano should ignore it when finding state for the next run but in this case the target may have explicitly emitted ` ` leading Meltano to store it and think it was valid for the next run

ripe-musician-59933 · Answer

Since that line I just linked to would emit ` ` ahead of any real state

ripe-musician-59933 · Answer

So I m thinking that s a bug in the target it shouldn t output any state until it gets a real state message from the tap

ripe-musician-59933 · Answer

Otherwise it s essentially resetting state entirely as you re seeing

curved-lunch-43500 · Answer

I didn t understand this message Since that line I just linked to would emit ` ` ahead of any real state What do you mean by I just linked

ripe-musician-59933 · Answer

I was referring to <https meltano slack com archives CFG3C3C66 p1614622019051000 thread ts=1614621579 049600 amp cid=CFG3C3C66>

curved-lunch-43500 · Answer

This is the log in the run that fails I thought it was emitting state

ripe-musician-59933 · Answer

The problem is that `target bigquery | INFO Pushing state ` line in your logs

curved-lunch-43500 · Answer

Ohh

curved-lunch-43500 · Answer

Got it now

ripe-musician-59933 · Answer

The target shouldn t be outputting any empty initial state it should wait for the first real state coming from the tap

ripe-musician-59933 · Answer

Apparently the target supports an `initial state` setting but if that isn t set instead of assuming an initial state of ` ` the target shouldn t emit an initial state at all

curved-lunch-43500 · Answer

Understood I ll open an issue in thir repo about it

ripe-musician-59933 · Answer

Sounds good

curved-lunch-43500 · Answer

Thanks a lot < ripe musician 59933> smile

ripe-musician-59933 · Answer

Glad I could help slightly smiling face