Does anyone know if there are any snowflake targets that will do append only replication? I’m writing a tap for a high volume graphql api. The tap pulls incrementally with a marker, but the API does not return the marker per row (only per call). There is no real primary key other than hashing the entire row. I don’t want to make that computation on millions of rows per day if there is another option.

Hi Stephan, unfortunately I haven't been able to find much documentation on the specifics of how the variants work. However, I did notice that the datamill variant uses an INSERT, which may be what you are looking for: https://meltano.com/plugins/loaders/snowflake.html / https://github.com/datamill-co/target-snowflake/blob/master/target_snowflake/snowflake.py#L395

thanks!

@stephen_lloyd Agreed. Many targets (including Snowflake) default to upsert if the have a primary key defined. When there is no primary key they should simply insert whatever records are sent by the tap. Since it sounds like your tap is only sending new records (using it's own internal method), I think you'll get the behavior you want automatically as long as you are using "incremental" replication method at the tap level.