Hi all, I am having a problem with incremental loading where all the data is still being updated, even though I have set the replication key and replication method to

incremental

on stream file and my meltano.yml file. Additionally, I am also calling the state when running meltano elt. Did I miss something? Fyi I am use custom tap which built by ex colleague.

There’s a couple of things to check: • That the tap declares the

state

capability in `meltano.yml`: https://github.com/edgarrmondragon/tap-bitso/blob/ed6d1696ddbd3b1c33897416f1453b673cdc4a82/meltano.yml#L10-L15 • That the tap calls get_starting_timestamp or get_starting_replication_key_value where the replication value is needed to filter results: https://github.com/edgarrmondragon/tap-bitso/blob/ed6d1696ddbd3b1c33897416f1453b673cdc4a82/tap_bitso/client.py#L87-L98

@edgar_ramirez_mondragon •

state

has been written on meltano.yml •

get_starting_timestamp

also has been impelemented on client.py as I used updated_at as replication_key a However, still it also run as full load. Do I miss something here ?

@ono which command are you running your pipeline with meltano?

meltano --log-level=debug --environment local elt tap-xxx target-postgres --state-id dev-test

I used this command.

ok, so you’re correctly using

--state-id

. If you run the tap with

meltano invoke tap-xxx

do you see a

{"type": "STATE", …}

message at the end?

@edgar_ramirez_mondragon thanks Edgar I fix it after I deeply read your previous code.

Awesome!