Morning all I m seeing a weird issue and wondering if its a Meltano #singer-target-development

Morning all - I'm seeing a weird issue and wonderi...

pnadolny

08/13/2021, 2:11 PM

Morning all - I'm seeing a weird issue and wondering if its a bug or something im doing wrong. I'm using a full sync from a snowflake view to my custom target built using the SDK and im running them using

meltano elt

. I just noticed in the system db that the state is being messed up. I'm also running a similar sync using meltano from snowflake to a different target not built with the SDK which is not having this state issue.

pnadolny

08/13/2021, 2:12 PM

after running once the state looks like this

{"singer_state": {"type": "STATE", "value": {"currently_syncing": *null*, "bookmarks": {"MY_SNOWFLAKE_DB-SCHEMA-TABLE_NAME": {"initial_full_table_complete": *true*}}}}}

pnadolny

08/13/2021, 2:13 PM

after each run its gets futher nested under another

{"type": "STATE", "value":

so the second run looks like this

{"singer_state": {"type": "STATE", "value": {"type": "STATE", "value": {"currently_syncing": *null*, "bookmarks": {"MY_SNOWFLAKE_DB-SCHEMA-TABLE_NAME": {"initial_full_table_complete": *true*}}}}}}

pnadolny

08/13/2021, 2:16 PM

it just continuously grows like that

pnadolny

08/13/2021, 2:17 PM

the state message for another target that I previously built not using the SDK but also being run with

meltano elt

im seeing a state message like

{"singer_state": {"currently_syncing": *null*, "bookmarks": {"MY_SNOWFLAKE_DB-SCHEMA-TABLE_NAME": {"initial_full_table_complete": *true*}}}}

which is correct

douwe_maan

08/13/2021, 2:34 PM

@pnadolny What target are you using? It looks like it may be outputting full ``{"type": "STATE", "value": <payload> }` messages, instead of just the payload

douwe_maan

08/13/2021, 2:34 PM

Ah, it’s a custom target built using the SDK

douwe_maan

08/13/2021, 2:34 PM

I wonder if this is a bug in the SDK cc @aaronsteers

pnadolny

08/13/2021, 2:39 PM

yes a custom target built with the SDK. I'm not seeing it with an old target build without the SDK but our new version with the SDK has it, so I'm trying to figure out if we did something wrong in our implementation or if its a bug in the SDK

douwe_maan

08/13/2021, 2:40 PM

I think this line is wrong: https://gitlab.com/meltano/sdk/-/blob/main/singer_sdk/target_base.py#L323

douwe_maan

08/13/2021, 2:40 PM

It’s supposed to just write