Happy Friday!
I did a search but I am not sure I a...
# troubleshootingb
barbarag
02/04/2022, 7:05 PMHappy Friday!
I did a search but I am not sure I am even looking in the right direction so 🤞
We run a nightly incremental extract with an itersize value of 20000. All of a sudden it is grabbing ~7k records and taking a minute for each batch. This is unacceptable given the size of the data. It is setting off alarms on the client’s source system. The only changes that have happened recently is that we added AWS Elastic File System to gather the data before moving it to s3 and a Meltano infrastructure upgrade. The upgrade took out our update date which caused an unexpected full run but we updated that and I don’t see how that would cause or continue to cause this.
Has anyone run into this? Or have ideas? Production is essentially down until we resolve this.
Copy code
WHERE "updated_at" >= '2022-01-31 07:45:06.807764'::timestamp without time zone
ORDER BY "updated_at" ASC with itersize 20000barbarag
02/04/2022, 7:06 PMwe are using tap-postgres. Forgot that tidbit.
v
visch
02/05/2022, 5:51 PM