Thanks for answering @visch 😄
Basically, I'm wondering:
• What is the use of using an SQL sink if you can use a record sink (what are the differences)
◦ Are record sinks made more general?
◦ Are SQLSinks made specifically for SQL type storage (bigquery, postgres, snowflake, etc.)
◦ What advantage / disadvantage is there to use one or another when pointing towards a SQL-type destination (Bigquery)
And then, some more general questions:
• How is _sdc_* information included in your sink? Does this behaviour have to be code, or is it included in the tap?
• What are the best practices between upsert & append? What is preferable?
My current goal is to create a target bigquery that:
• Supports log replication, incremental replication & full table
• Include _sdc information