For taps built with the SDK, how is the batch size...
# singer-tapsc
Conner Panarella (SpaceCondor)
09/12/2024, 8:47 PMFor taps built with the SDK, how is the batch size configured? It seems to be 10,000 but is this configurable at all?
r
Reuben (Matatika)
09/12/2024, 9:33 PMNot for a tap. For a target built with the SDK, yes with `batch_size_rows`:
https://github.com/meltano/sdk/blob/170f5bc512aab088560d90e6c646c53be30986c9/singer_sdk/helpers/capabilities.py#L155-L161
https://github.com/meltano/sdk/blob/170f5bc512aab088560d90e6c646c53be30986c9/singer_sdk/sinks/core.py#L192-L194
Reuben (Matatika)
09/12/2024, 9:40 PMHmm... Maybe with
batch_config.batch_sizee
Edgar RamĂrez (Arch.dev)
09/12/2024, 10:15 PMBatch is an overloaded concept here, so I'm curious what's the problem you're trying to solve @Conner Panarella (SpaceCondor) 🙂
c
Conner Panarella (SpaceCondor)
09/12/2024, 10:36 PM@Edgar RamĂrez (Arch.dev) Mainly just testing! I want to optimize the loading of 1m+ records from a MSSQL tap to the Postgres loader.
Conner Panarella (SpaceCondor)
09/12/2024, 10:38 PMThe BATCH functionality documentation doesn’t really explain how it works between taps and targets.
18 Views