Good morning, I'm seeking general guidance on opti...
# troubleshooting
m
Good morning, I'm seeking general guidance on options for performance improvements with Meltano. • I already found this blogpost about BATCH-Messages: https://meltano.com/blog/6x-more-speed-for-your-data-pipelines-with-batch-messages/ • Is there any way to provide more ressources to Meltano via the config? • Is there any other option if the data might be too big? Thank you & kind regards, Marc
e
BATCH messages is a good start.
• Is there any way to provide more ressources to Meltano via the config?
• Is there any other option if the data might be too big?
It depends on the extractor and loader, you'll want to increase the batch size via config and also the memory of your instance/container
m
Thanks Edgar, I'll take a deeper look at it