I was using a target and figured it will be useful...
# troubleshootingp
prakhar_srivastava
10/17/2023, 3:06 PMI was using a target and figured it will be useful to have batch_numbers; if the source and filter is same batch number can provide idempotency for downstream tasks but there is only batch_id and batch_time both of which are generated on runtime hence for the same source and same filter we get different batch config even if the batch is identical, it this solved or discussed?
prakhar_srivastava
10/17/2023, 3:15 PMI think I can quickly do something like:
context["batch_number"] = self._total_records_written / self.max_sizeprakhar_srivastava
10/17/2023, 3:16 PMin the
process_batchprakhar_srivastava
10/17/2023, 3:17 PMbut would be a good add
prakhar_srivastava
10/17/2023, 3:18 PMespecially useful to stream which have replication
prakhar_srivastava
10/17/2023, 4:37 PMthis doesn't work
2 Views