dan_ladd
10/04/2022, 2:29 PMtarget-salesforce
š¤
I'm debating how to handle update/insert/upsert/delete/hard deletes. Each type is a different API call with simple-salesforce
⢠Does it make sense to have the streams segmented by type?
⦠Example Account-delete
, Account
(default to update), Account-upsert
....
⢠Or do I keep all the records in the Account
stream and allow for an optional _type
field to dictate the behavior? This seems cleaner.
⦠This is easy if I write each record individually, but not sure about sub-segmenting a batch sink.
Any ideas? Is there a target that has already addressed something similar?visch
10/04/2022, 4:46 PMAccount
stream, with a action
column (probably a better name would be _sdc_action
) with values of insert
update
delete
The thing that would be very helpful to have in these types of targets is a schema that the stream should conform to. Wouldn't be that crazy, I just think the record should have to conform to the schema Account
accepts. Right now I just do record["name"]
, record["id"]
etc etc, and you can run into typing issues with that.visch
10/04/2022, 4:47 PMvisch
10/04/2022, 4:47 PMvisch
10/04/2022, 4:50 PMdan_ladd
10/04/2022, 4:52 PMdan_ladd
10/04/2022, 4:55 PM_sdc_action
route and batch them in their own Account-delete.jsonl
Account-update.jsonl
... files and then write the files when reaching 10k Account records.visch
10/04/2022, 4:59 PM.jsonl
file and then reparse the file but what do I knowdan_ladd
10/04/2022, 5:00 PMvisch
10/04/2022, 5:00 PMvisch
10/04/2022, 5:01 PMdan_ladd
10/04/2022, 5:14 PMvisch
10/04/2022, 5:22 PMaaronsteers
10/04/2022, 10:20 PM_sdc_deleted_at
property which signifies a record was or should be deleted.aaronsteers
10/04/2022, 10:21 PMaaronsteers
10/04/2022, 10:22 PMvisch
10/05/2022, 1:01 AM