Hi community! I've a question about states after a...
# announcementsm
murilo_tavolaro_de_nigris
03/01/2021, 5:59 PMHi community! I've a question about states after a job failed using Adswerve/target-bigquery.
After a tap fails, the next run uses the START_DATE set in the config variable, instead of picking up the state that failed.
I saw this note in the documentation (in the end of the page) that some loaders might interrupt the pipeline before meltano writes the state. Does this happen with the adswerve/target-bigquery?
d
douwe_maan
03/01/2021, 6:06 PM@murilo_tavolaro_de_nigris Reading through https://github.com/adswerve/target-bigquery/blob/master/target_bigquery/processhandler.py, it appears to always emit state whenever a STATE message comes in from the tap: https://github.com/adswerve/target-bigquery/blob/master/target_bigquery/processhandler.py#L253, and whenever it's done processing a stream: https://github.com/adswerve/target-bigquery/blob/master/target_bigquery/processhandler.py#L262
douwe_maan
03/01/2021, 6:06 PMAlso "yield init state, so even if there is an exception right after we get proper state emitted": https://github.com/adswerve/target-bigquery/blob/master/target_bigquery/process.py#L22
douwe_maan
03/01/2021, 6:08 PMBut there may not be any initial state if the target fails pretty much immediately
douwe_maan
03/01/2021, 6:10 PMIf the failed pipeline run didn't emit any state, Meltano should ignore it when finding state for the next run, but in this case the target may have explicitly emitted
{}douwe_maan
03/01/2021, 6:10 PMSince that line I just linked to would emit
{}douwe_maan
03/01/2021, 6:11 PMSo I'm thinking that's a bug in the target: it shouldn't output any state until it gets a real state message from the tap
douwe_maan
03/01/2021, 6:11 PMOtherwise it's essentially resetting state entirely, as you're seeing
m
murilo_tavolaro_de_nigris
03/01/2021, 6:14 PMI didn't understand this message:
"Since that line I just linked to would emit
{}d
douwe_maan
03/01/2021, 6:14 PMm
murilo_tavolaro_de_nigris
03/01/2021, 6:14 PMThis is the log in the run that fails, I thought it was emitting state
d
douwe_maan
03/01/2021, 6:14 PMThe problem is that
target-bigquery | INFO Pushing state: {}m
murilo_tavolaro_de_nigris
03/01/2021, 6:15 PMOhh
murilo_tavolaro_de_nigris
03/01/2021, 6:15 PMGot it now!!
d
douwe_maan
03/01/2021, 6:15 PMThe target shouldn't be outputting any empty initial state, it should wait for the first real state coming from the tap
douwe_maan
03/01/2021, 6:16 PMApparently the target supports an
initial_state{}m
murilo_tavolaro_de_nigris
03/01/2021, 6:16 PMUnderstood, I'll open an issue in thir repo about it.
d
douwe_maan
03/01/2021, 6:16 PMSounds good!
m
murilo_tavolaro_de_nigris
03/01/2021, 6:16 PMThanks a lot @douwe_maan 😄
d
douwe_maan
03/01/2021, 6:16 PMGlad I could help 🙂
10 Views