I'm having a performance issue cross-region. I'm using meltanolabs tap-postgres to copy from Postgres in US to Redshift in Australia. In my logs I can see there's a delay between when the tap starts and when it starts doing something.

735
2024-09-03T13:58:19.094+10:00
2024-09-03 03:58:19,094 | INFO | tap-postgres.public-forms | Beginning incremental sync of 'public-forms'...
736
2024-09-03T13:58:19.093+10:00
2024-09-03 03:58:19,093 | INFO | tap-postgres | Skipping deselected stream 'public-audit_collection_events'.
737
2024-09-03T13:57:02.987+10:00
Environment 'us' is active
738
2024-09-03T13:57:01.573+10:00
running tap [tap-agnodice-v3] with mappers: [mapper-agnodice] to target [target-warehouse-redshift]
739
2024-09-03T13:57:01.573+10:00
Executing: pipenv run .meltano/run/bin --environment=us run --no-install tap-agnodice-v3 mapper-agnodice target-warehouse-redshift

It's about 80 seconds there, but I've seen it up to 5 mins. Any ideas how to improve this?

My guess would be something related to catalog discovery, maybe if there are a lot of tables or particularly large ones being processed. Some things to try: • Select only the tables you are interested in

# select only the public-forms table
select:
- public-forms.*

• Have a look at the

filter_schemas

setting if you have more than one schema • Provide your own catalog to bypass the discovery process, if you know the structure of the data you expect - bear in mind if you do this, you are in full manual control of the catalog, so concepts like

select

no longer apply https://docs.meltano.com/guide/integration/#extractor-catalog-generation https://docs.meltano.com/concepts/plugins/#catalog-extra