Hey guys has anyone encountered the psycopg out of memory er Meltano #singer-targets

Hey guys, has anyone encountered the psycopg out o...

Matthew Podolak

11/25/2024, 4:11 PM

Hey guys, has anyone encountered the psycopg out of memory error with the postgres target? This is my meltano config:

Copy code

version: 1
default_environment: prod
project_id: ownr-wh-meltano
environments:
  - name: prod

database_uri: $DATABASE_URL?options=-csearch_path%3Dmeltano

plugins:
  extractors:
  - name: tap-postgres
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/tap-postgres.git>
    select:
      - '!public-pg_stat_statements.*'
      - '!public-pg_stat_statements_info.*'
      - public-*.*
  - name: tap-sage-db
    inherit_from: tap-postgres
    config:
      sqlalchemy_url: $SAGE_DB_URL
      default_replication_method: INCREMENTAL
      filter_schemas: 
        - public
    metadata:
      '*':
        replication-key: updated_at
      '*campaign_history*':
        replication-key: id
      '*email_history':
        replication-key: id
      '*leaving_reasons':
        replication-key: id
  loaders:
  - name: target-postgres
    variant: meltanolabs
    pip_url: meltanolabs-target-postgres
    config:
      sqlalchemy_url: $DATABASE_URL
      load_method: upsert
      validate_records: false
  - name: target-sage-warehouse-schema
    inherit_from: target-postgres
    config:
      default_target_schema: sage
      batch_size_rows: 5000

👀 1

Edgar Ramírez (Arch.dev)

11/25/2024, 4:16 PM

you can try using a lower `batch_size_rows`:

Copy code

config:
  batch_size_rows: 5000

The default is

, but I think we don't document that default value anywhere. We should.

Matthew Podolak

11/25/2024, 4:29 PM

I'll give it a try, it seems like meltano is running a lot of concurrent streams as well which probably contributes to the error, and I dont see any ways to reduce this

Matthew Podolak

11/25/2024, 8:50 PM

@Edgar Ramírez (Arch.dev) I tried with the batch size of 1000 and repeatedly got the out of memory error. Im trying to replicate millions of rows, so I dont want to go any smaller. Do you know of any other memory management techniques with meltano?

Edgar Ramírez (Arch.dev)

11/25/2024, 9:12 PM

Ok, can you share some more details about your setup? • the command you're running • the traceback/exception revealing the memory error • any other details you think might be relevant another option worth trying, which is still in preview at this point is

Copy code

loaders:
  - name: target-postgres
    variant: meltanolabs
    pip_url: meltanolabs-target-postgres==0.2.0a3
    config:
      use_copy: true
      ... # your other settings here

Matthew Podolak

11/26/2024, 2:02 AM

Copy code

Nov 25 15:24:15 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": "2024-11-25 20:24:15,038 | INFO | target-postgres | Target sink for 'public-matters' is full. Current size is '1000'. Draining...", "level": "info", "timestamp": "2024-11-25T20:24:15.038349Z"}
Nov 25 15:24:15 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": "2024-11-25 20:24:15,071 | INFO | target-postgres.public-matters | Inserting with SQL: INSERT INTO \"5326548f_5c0b_4576_97a4_e3a9b96384a5\" (id, created_at, updated_at, companyid, receipt, type, details, annullable, state, process_flow, _sdc_extracted_at, _sdc_received_at, _sdc_batched_at, _sdc_deleted_at, _sdc_sequence, _sdc_table_version, _sdc_sync_started_at) VALUES (:id, :created_at, :updated_at, :companyid, :receipt, :type, :details, :annullable, :state, :process_flow, :_sdc_extracted_at, :_sdc_received_at, :_sdc_batched_at, :_sdc_deleted_at, :_sdc_sequence, :_sdc_table_version, :_sdc_sync_started_at)", "level": "info", "timestamp": "2024-11-25T20:24:15.072545Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": "2024-11-25 20:24:18,561 | INFO | singer_sdk.metrics | METRIC: {\"type\": \"timer\", \"metric\": \"batch_processing_time\", \"value\": 3.52325439453125, \"tags\": {\"stream\": \"public-matters\", \"pid\": 234, \"status\": \"failed\"}}", "level": "info", "timestamp": "2024-11-25T20:24:18.561817Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": "Traceback (most recent call last):", "level": "info", "timestamp": "2024-11-25T20:24:18.562166Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " File \"/app/.meltano/loaders/target-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py\", line 2118, in _exec_insertmany_context", "level": "info", "timestamp": "2024-11-25T20:24:18.562330Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " dialect.do_execute(", "level": "info", "timestamp": "2024-11-25T20:24:18.562532Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " File \"/app/.meltano/loaders/target-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/default.py\", line 941, in do_execute", "level": "info", "timestamp": "2024-11-25T20:24:18.562671Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " cursor.execute(statement, parameters)", "level": "info", "timestamp": "2024-11-25T20:24:18.562797Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": "psycopg2.errors.OutOfMemory: out of memory", "level": "info", "timestamp": "2024-11-25T20:24:18.562920Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": "DETAIL: Failed on request of size 524288 in memory context \"ErrorContext\".", "level": "info", "timestamp": "2024-11-25T20:24:18.563040Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": "", "level": "info", "timestamp": "2024-11-25T20:24:18.563173Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": "", "level": "info", "timestamp": "2024-11-25T20:24:18.563295Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": "The above exception was the direct cause of the following exception:", "level": "info", "timestamp": "2024-11-25T20:24:18.563416Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": "", "level": "info", "timestamp": "2024-11-25T20:24:18.563536Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": "Traceback (most recent call last):", "level": "info", "timestamp": "2024-11-25T20:24:18.563657Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " File \"/app/.meltano/loaders/target-postgres/venv/bin/target-postgres\", line 8, in <module>", "level": "info", "timestamp": "2024-11-25T20:24:18.563779Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " sys.exit(TargetPostgres.cli())", "level": "info", "timestamp": "2024-11-25T20:24:18.563902Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " File \"/app/.meltano/loaders/target-postgres/venv/lib/python3.9/site-packages/click/core.py\", line 1157, in __call__", "level": "info", "timestamp": "2024-11-25T20:24:18.564028Z"}

Matthew Podolak

11/26/2024, 2:02 AM

Copy code

Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " return self.main(*args, **kwargs)", "level": "info", "timestamp": "2024-11-25T20:24:18.564154Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " File \"/app/.meltano/loaders/target-postgres/venv/lib/python3.9/site-packages/click/core.py\", line 1078, in main", "level": "info", "timestamp": "2024-11-25T20:24:18.564276Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " rv = self.invoke(ctx)", "level": "info", "timestamp": "2024-11-25T20:24:18.564398Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " File \"/app/.meltano/loaders/target-postgres/venv/lib/python3.9/site-packages/singer_sdk/plugin_base.py\", line 84, in invoke", "level": "info", "timestamp": "2024-11-25T20:24:18.564523Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " return super().invoke(ctx)", "level": "info", "timestamp": "2024-11-25T20:24:18.564645Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " File \"/app/.meltano/loaders/target-postgres/venv/lib/python3.9/site-packages/click/core.py\", line 1434, in invoke", "level": "info", "timestamp": "2024-11-25T20:24:18.564767Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " return ctx.invoke(self.callback, **ctx.params)", "level": "info", "timestamp": "2024-11-25T20:24:18.564939Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " File \"/app/.meltano/loaders/target-postgres/venv/lib/python3.9/site-packages/click/core.py\", line 783, in invoke", "level": "info", "timestamp": "2024-11-25T20:24:18.565071Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " return __callback(*args, **kwargs)", "level": "info", "timestamp": "2024-11-25T20:24:18.565194Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " File \"/app/.meltano/loaders/target-postgres/venv/lib/python3.9/site-packages/singer_sdk/target_base.py\", line 572, in invoke", "level": "info", "timestamp": "2024-11-25T20:24:18.565317Z"}

Matthew Podolak

11/26/2024, 2:02 AM

Copy code

Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " target.listen(file_input)", "level": "info", "timestamp": "2024-11-25T20:24:18.565446Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " File \"/app/.meltano/loaders/target-postgres/venv/lib/python3.9/site-packages/singer_sdk/_singerlib/encoding/_base.py\", line 48, in listen", "level": "info", "timestamp": "2024-11-25T20:24:18.565571Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " self._process_lines(file_input or self.default_input)", "level": "info", "timestamp": "2024-11-25T20:24:18.565694Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " File \"/app/.meltano/loaders/target-postgres/venv/lib/python3.9/site-packages/singer_sdk/target_base.py\", line 304, in _process_lines", "level": "info", "timestamp": "2024-11-25T20:24:18.565814Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " counter = super()._process_lines(file_input)", "level": "info", "timestamp": "2024-11-25T20:24:18.565965Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " File \"/app/.meltano/loaders/target-postgres/venv/lib/python3.9/site-packages/singer_sdk/_singerlib/encoding/_base.py\", line 70, in _process_lines", "level": "info", "timestamp": "2024-11-25T20:24:18.566089Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " self._process_record_message(line_dict)", "level": "info", "timestamp": "2024-11-25T20:24:18.566213Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " File \"/app/.meltano/loaders/target-postgres/venv/lib/python3.9/site-packages/singer_sdk/target_base.py\", line 371, in _process_record_message", "level": "info", "timestamp": "2024-11-25T20:24:18.566337Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " self.drain_one(sink)", "level": "info", "timestamp": "2024-11-25T20:24:18.566460Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " File \"/app/.meltano/loaders/target-postgres/venv/lib/python3.9/site-packages/singer_sdk/target_base.py\", line 517, in drain_one", "level": "info", "timestamp": "2024-11-25T20:24:18.566585Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " sink.process_batch(draining_status)", "level": "info", "timestamp": "2024-11-25T20:24:18.566707Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " File \"/app/.meltano/loaders/target-postgres/venv/lib/python3.9/site-packages/target_postgres/sinks.py\", line 95, in process_batch", "level": "info", "timestamp": "2024-11-25T20:24:18.566829Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " self.bulk_insert_records(", "level": "info", "timestamp": "2024-11-25T20:24:18.566955Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " File \"/app/.meltano/loaders/target-postgres/venv/lib/python3.9/site-packages/target_postgres/sinks.py\", line 176, in bulk_insert_records", "level": "info", "timestamp": "2024-11-25T20:24:18.567080Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " connection.execute(insert, data_to_insert)", "level": "info", "timestamp": "2024-11-25T20:24:18.567215Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " File \"/app/.meltano/loaders/target-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py\", line 1418, in execute", "level": "info", "timestamp": "2024-11-25T20:24:18.567338Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " return meth(", "level": "info", "timestamp": "2024-11-25T20:24:18.567459Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " File \"/app/.meltano/loaders/target-postgres/venv/lib/python3.9/site-packages/sqlalchemy/sql/elements.py\", line 515, in _execute_on_connection", "level": "info", "timestamp": "2024-11-25T20:24:18.567579Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " return connection._execute_clauseelement(", "level": "info", "timestamp": "2024-11-25T20:24:18.567701Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " File \"/app/.meltano/loaders/target-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py\", line 1640, in _execute_clauseelement", "level": "info", "timestamp": "2024-11-25T20:24:18.567821Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " ret = self._execute_context(", "level": "info", "timestamp": "2024-11-25T20:24:18.567941Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " File \"/app/.meltano/loaders/target-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py\", line 1844, in _execute_context", "level": "info", "timestamp": "2024-11-25T20:24:18.568061Z"}

Matthew Podolak

11/26/2024, 2:02 AM

Copy code

Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " return self._exec_insertmany_context(dialect, context)", "level": "info", "timestamp": "2024-11-25T20:24:18.568181Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " File \"/app/.meltano/loaders/target-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py\", line 2126, in _exec_insertmany_context", "level": "info", "timestamp": "2024-11-25T20:24:18.568305Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " self._handle_dbapi_exception(", "level": "info", "timestamp": "2024-11-25T20:24:18.568427Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " File \"/app/.meltano/loaders/target-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py\", line 2355, in _handle_dbapi_exception", "level": "info", "timestamp": "2024-11-25T20:24:18.568556Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema", "cmd_type": "elb", "run_id": "511cb501-7365-4c9c-8cbe-c81dfa3c813c", "job_name": "prod:tap-sage-db-to-target-sage-warehouse-schema", "stdio": "stderr", "name": "target-sage-warehouse-schema", "event": " raise sqlalchemy_exception.with_traceback(exc_info[2]) from e", "level": "info", "timestamp": "2024-11-25T20:24:18.568722Z"}
Nov 25 15:24:18 data-warehouse-shield-prod app[cron-python] info {"consumer": true, "producer": false, "string_id": "target-sage-warehouse-schema",

Matthew Podolak

11/26/2024, 2:04 AM

command:

meltano run tap-sage-db target-sage-warehouse-schema --force

Matthew Podolak

11/26/2024, 2:05 AM

meltano==3.5.4

python-3.9.16

Matthew Podolak

11/26/2024, 2:06 AM

how would the

use_copy

config help avoid the DB memory issues? are there any other options for memory management in meltano? or how is this handled by default - i cant seem to find any info on it unlike similar libraries like

dlt

35 Views

Open in Slack

Previous Next