My uneducated guess is that the state emitted from...
# troubleshooting
d
My uneducated guess is that the state emitted from the full table sync needs to be merged with the state from logical sync rather than overwritten, but not sure really, I still don’t have a great grasp on how meltano and the singer spec interact
a
@dean_morin - Actually - yes, I think you actually may have nailed it here. The tap should be keeping its state and emitting a cumulative state downstream, but as you point out, it appears there's a "reset" between modes, which is causing the first state message to get clobbered by the second. Since the second and final state message you show does not have any entry for
public-b
, I think that's causing the issue you are seeing.
@douwe_maan - Do you agree? From the logs here above in the channel, the second and final version of the state message should be a superset of streams sent so far in this invocation.
d
Yeah, looks like a bug where the tap doesn’t handle state correctly when some streams are doing a full-table sync and others are already on log-based
d
Thanks! Since it sounds like it’s a tap bug, I’ll post on the transferwise singer slack channel and see what they say
It’s definitely a bug with the target (reproduced it with plain ol’ singer). Opened an issue here: https://github.com/transferwise/pipelinewise-target-snowflake/issues/192
Actually come to think of it, maybe it should be the tap managing this state…