I’ve run into an interesting behavior with

pipelinewise-target-redshift

where a pipeline will run just fine for quite a while, then fail because of an apparent issue reading a bookmark out of the state. Here’s a snippet of the logs:

pipelinewise-target-redshift |     if stream in state.get('bookmarks', {}):
pipelinewise-target-redshift | AttributeError: 'NoneType' object has no attribute 'get'
meltano                      | [Errno 32] Broken pipe

Has anyone else run into this? So far I’ve only seen it happen with

tap-mailchimp

, but it seems to be inconsistent. I’ll load a particular table and it’ll run fine, but may fail on a subsequent reload.

Hey! I haven't run into this error and pipelinewise-target-redshift was working fine on my end (other than the fact that it doesn't handle multi-level nested columns). However, I will let you know if I find anything in regards to a solution!

On that front - I’ve had some success loading JSON-y columns as is and then unpacking once they’re loaded via DBT if that’s an option for you?

@nick_hamlin Judging from the pipelinewise-target-redshift code, it looks like it could fail if the tap never sends a

STATE

message (which would result in the

state

variable never getting a value other than

None

), but tap-mailchimp does write state under various circumstances, e.g. https://github.com/singer-io/tap-mailchimp/blob/af00aa1a421221e8df536dfbb4f8f87b01e1ef85/tap_mailchimp/sync.py#L142. But perhaps when you're seeing the error, it's not. You could confirm this by running the pipeline with

meltano --log-level=debug elt

which will show you any STATE message that the tap (doesn't) emit. It looks like enabling the

flush_all_streams

setting on the target would work around the offending code path: https://github.com/transferwise/pipelinewise-target-redshift/blob/12f79884f4d5947e7c1647b87a555bfeff625771/target_redshift/__init__.py#L182 Either way I'd consider this a bug in the target that you may want to fix; it shouldn't fail when the tap doesn't output state.

Interesting, thanks for the point in the right direction @douwe_maan! I wonder if perhaps there’s some quirk with tap-mailchimp where it’s not writing that state everytime, but all my other taps are (hence why I’m only seeing this periodically, and only with the MC tap). I’ll take a look