Hi, I would like to initially sync the data from postgres db (using meltano / transferwise variants of

tap-postgres

), but I get the following issue:

The extractor generated a message exceeding the message size limit of 5.0MiB (half the buffer size of 10.0MiB).

How to overcome it? I would need some iteration that I sync every 20k or 100k rows, because the table has couple of mils. Or it is already being done, but should I look for the issue inside table that I can't even sync 20k rows, because the size my table of 20k rows is too big (more than 5mb)

https://docs.meltano.com/reference/settings/#eltbuffer_size needs to be more easily searchable based on the error, I thought this error included the workaround though

I played around with this config and set to 25MiB, but the error is the same. Maybe I need to make it 50?

What did you set the config value to, your error says you have it set to 10MiB

That is the default. I then set to 25MiB and got similar error, just saying

The extractor generated a message exceeding the message size limit of 12.5MiB (half the buffer size of 25.0MiB).

Should I try increase it? 🙂

yes

Do you know how many rows should be synced by default using https://hub.meltano.com/extractors/tap-postgres--meltanolabs/, because with transferwise version I see it is 20000?