Hello everyone wave A potential misunderstanding from my end Meltano #troubleshooting

Hello everyone :wave: A potential misunderstanding...

vincent_legendre

09/05/2022, 2:10 PM

Hello everyone 👋 A potential misunderstanding from my end here : I cannot get my extract-load pipeline to stop on its own using the documented tap-postgres params

max_run_seconds.

I may precise that I am performing the first sync for a big table, which would need a large number of runs to get completed. My meltano config is in thread.

vincent_legendre

09/05/2022, 2:15 PM

Copy code

- name: tap-postgres
    config:
      host: my-host
      port: 5432
      user: user
      dbname: db
      filter_schemas: my-schema
      itersize: 1000
      max_run_seconds: 30
  loaders:
  - name: target-bigquery
    variant: adswerve
    pip_url: git+<https://github.com/adswerve/target-bigquery.git@0.12.0>
    config:
      add_metadata_columns: true
      dataset_id: meltano
      location: europe-west1
      replication_method: append
      credentials_path: load/bigquery/creds.json
      table_config: load/bigquery/target-tables-config.json

christoph

09/05/2022, 10:16 PM

max_run_seconds

is a setting for the logical replication method. https://github.com/transferwise/pipelinewise-tap-postgres/blob/da2fad9fd6efea96f83[…]c0d54a724f2/tap_postgres/sync_strategies/logical_replication.py As you are doing an initial sync, meltano will call the tap in full table mode, not log-based mode.

vincent_legendre

09/06/2022, 7:29 AM

Thanks for your answer Christoph 🙂 Is there is a way to control how Meltano performs full table sync nonetheless? Our infrastructure imposes on us to be precautionous with resources usage

christoph

09/06/2022, 8:07 AM

I see. Could you clarify what those precautions are? My assumption would be that your source table is getting written to a lot all the time and those 'insert' and 'update' transactions need to be prevented from failing? Am I in the ballpark with that assumption?

vincent_legendre

09/06/2022, 8:46 AM

Our postgres cluster has a master/slave configuration, so we are quite safe regarding these transactions. However, a large a number of pipelines are based on the slave instance, and we do not want to lower their performances when replicating data to BigQuery. Another important concern is bandwith usage, especially for outbound traffic. The approach we had until now was to orchestrate our extractions with Airflow (using PythonOperators), and to cap execution parallelism using Airflow parameters.

Open in Slack

Previous Next