When developing a tap if one of the streams is `FULL TABLE` Meltano #announcements

When developing a tap, if one of the streams is `F...

nehil_jain

09/21/2020, 2:45 PM

When developing a tap, if one of the streams is

FULL_TABLE

https://github.com/singer-io/getting-started/blob/master/docs/SYNC_MODE.md#replication-method. Is there something special the tap has to do to notify the target that this stream is full table and not incremental. How does the target know if the new rows are supposed to be appended or (delete + insert)?

douwe_maan

09/21/2020, 3:05 PM

@nehil_jain The replication method only affects the tap's behavior, not the target's. It's a way of configuring per-stream whether to "use state and only sync new records" (

INCREMENTAL

) or "ignore state and always go back to `start_date`" (

FULL_TABLE

). The target always behaves identically with regard to any records it's fed by the tap

douwe_maan

09/21/2020, 3:07 PM

It's up to the target to use the schema's

key_properties

to find the record's key properties (e.g.

id

) and decide whether to insert or update (upsert), depending on whether a record with that key already exists in the destination

nehil_jain

09/21/2020, 3:14 PM

aah, makes sense.

jeff_huth__bytecode.io_

09/26/2020, 7:34 PM

Most taps are upsert only; either FULL_TABLE (all records) or INCREMENTAL (only records on/after the bookemark). But there are a few taps out there that use

ACTIVATE_VERSION

. This is an undocumented feature and only works if the target is set up to deal with this... basically, it loads ALL records with a version, then (logically?) deletes earlier version(s) of the records with the same Primary Key. These allow you to upsert + delete (to logically delete the "delta" records). Imagine you load 100 records from a table today; then you delete 5 records and load 95 records tomorrow. This feature "deletes" the delta 5 records. Most database targets allow for this. But some file-based targets and even singer-check-tap target fail when you try to run these taps.