hello all, I’m trying to have a meltano ELT pipeli...
# troubleshootingj
Justin Yang
05/08/2025, 2:12 PMhello all, I’m trying to have a meltano ELT pipeline tapping Postgres and targeting Redshift. When I run
meltano run tap-postgres target-redshift Copy code
redshift_connector.error.InterfaceError: {'S': 'FATAL', 'C': '28000', 'M': 'no pg_hba.conf entry for host "???", user "wave", database "scylla", SSL off', 'F': '/opt/brazil-pkg-cache/packages/RedshiftPADB/RedshiftPADB-1.0.12086.0/AL2_x86_64/generic-flavor/src/src/pg/src/backend/libpq/auth.c', 'L': '477', 'R': 'ClientAuthentication'} cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=d990a55c-30b3-41b2-8bc0-2e20a200ee56 stdio=stderr string_id=target-redshift Copy code
Found credentials from IAM Role: Wave__DataEC2Role👀 1
✅ 1
e
Edgar Ramírez (Arch.dev)
05/08/2025, 3:38 PMWhich configs have you set for the target?
j
Justin Yang
05/08/2025, 5:42 PM Copy code
loaders:
- name: target-redshift
variant: ticketswap
pip_url: git+<https://github.com/TicketSwap/target-redshift.git>
config:
dbname: scylla
default_target_schema: meltano_ingestion
host: <host>
load_method: upsert
user: <user>
password: ${TARGET_REDSHIFT_PASSWORD}
s3_bucket: <bucket>
s3_prefix: ${TARGET_REDSHIFT_S3_KEY_PREFIX}
aws_redshift_copy_role_arn: <arn>
port: '5439'e
Edgar Ramírez (Arch.dev)
05/08/2025, 5:43 PMOk, so maybe something's off with the role being used to COPY files from S3?
j
Justin Yang
05/08/2025, 6:05 PMso one interesting point is that I’m running meltano on an EC2 instance. That ec2 instance has an IAM role connected to it but it is not pulling in the one that I specified in
aws_redshift_copy_role_arn: <arn>Justin Yang
05/08/2025, 6:05 PMalthough you bring a good point. What I can do is up the permissions for the one its trying to use
👍 1
Justin Yang
05/08/2025, 6:52 PMhey so I found the role it’s using, pretty much gave it full access to everything related - s3, ec2, redshift and gave extra trust policies but i’m still getting the same error
e
Edgar Ramírez (Arch.dev)
05/08/2025, 7:06 PMTry adding this config:
enable_iam_authentication: truej
Justin Yang
05/08/2025, 7:07 PMoh that worked, I think theres a different error now
Justin Yang
05/08/2025, 7:08 PMI can try some other way although is it possible to use credentials to access the redshift cluster?
e
Edgar Ramírez (Arch.dev)
05/08/2025, 7:12 PMYeah, I see you already have a user and password. Hmm.
Do you have the full traceback for the
redshift_connector.error.InterfaceErrorj
Justin Yang
05/08/2025, 7:38 PM Copy code
2025-05-08T19:38:44.161746Z [error ]
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /usr/local/lib/python3.12/site-packages/meltano/core/logging/output_logger.py:217 in │
│ redirect_logging │
│ │
│ 214 │ │ │ *ignore_errors, │
│ 215 │ │ ) │
│ 216 │ │ try: │
│ ❱ 217 │ │ │ yield │
│ 218 │ │ except ignored_errors: │
│ 219 │ │ │ raise │
│ 220 │ │ except RunnerError as err: │
│ │
│ /usr/local/lib/python3.12/site-packages/meltano/core/block/extract_load.py:509 in run │
│ │
│ 506 │ │ │ # TODO: legacy `meltano elt` style logging should be deprecated │
│ 507 │ │ │ legacy_log_handler = self.output_logger.out("meltano", logger) │
│ 508 │ │ │ with legacy_log_handler.redirect_logging(): │
│ ❱ 509 │ │ │ │ await self.run_with_job() │
│ 510 │ │ │ │ return │
│ 511 │ │ else: │
│ 512 │ │ │ logger.warning( │
│ │
│ /usr/local/lib/python3.12/site-packages/meltano/core/block/extract_load.py:549 in run_with_job │
│ │
│ 546 │ │ │
│ 547 │ │ with closing(self.context.session) as session: │
│ 548 │ │ │ async with job.run(session): │
│ ❱ 549 │ │ │ │ await self.execute() │
│ 550 │ │
│ 551 │ async def terminate(self, *, graceful: bool = False) -> None: │
│ 552 │ │ """Terminate an in flight ExtractLoad execution, potentially disruptive. │
│ │
│ /usr/local/lib/python3.12/site-packages/meltano/core/block/extract_load.py:501 in execute │
│ │
│ 498 │ │ async with self._start_blocks(): │
│ 499 │ │ │ await self._link_io() │
│ 500 │ │ │ manager = ELBExecutionManager(self) │
│ ❱ 501 │ │ │ await manager.run() │
│ 502 │ │
│ 503 │ async def run(self) -> None: │
│ 504 │ │ """Run the ELT task.""" │
│ │
│ /usr/local/lib/python3.12/site-packages/meltano/core/block/extract_load.py:716 in run │
│ │
│ 713 │ │ for IO to complete as appropriate. Expect a RunnerError to be raised if │
│ 714 │ │ any of the blocks exit with a non 0 exit code. │
│ 715 │ │ """ │
│ ❱ 716 │ │ await self._wait_for_process_completion(self.elb.head) │
│ 717 │ │ _check_exit_codes( │
│ 718 │ │ │ self._producer_code, │
│ 719 │ │ │ self._consumer_code, │
│ │
│ /usr/local/lib/python3.12/site-packages/meltano/core/block/extract_load.py:789 in │
│ _wait_for_process_completion │
│ │
│ 786 │ │ │ │ │ line_length_limit=self.line_length_limit, │
│ 787 │ │ │ │ │ stream_buffer_size=self.stream_buffer_size, │
│ 788 │ │ │ │ ) │
│ ❱ 789 │ │ │ raise output_futures_failed.exception() │
│ 790 │ │ │
│ 791 │ │ # If all the output handlers completed without raising an │
│ 792 │ │ # exception, we still need to wait for all the underlying block │
│ │
│ /usr/local/lib/python3.12/site-packages/meltano/core/logging/utils.py:305 in │
│ capture_subprocess_output │
│ │
│ 302 │ │ │ continue │
│ 303 │ │ │
│ 304 │ │ for writer in line_writers: │
│ ❱ 305 │ │ │ if not await _write_line_writer(writer, line): │
│ 306 │ │ │ │ # If the destination stream is closed, we can stop capturing output. │
│ 307 │ │ │ │ return │
│ 308 │
│ │
│ /usr/local/lib/python3.12/site-packages/meltano/core/logging/utils.py:273 in _write_line_writer │
│ │
│ 270 │ │ │ writer.write(line) │
│ 271 │ │ │ await writer.drain() │
│ 272 │ │ except (BrokenPipeError, ConnectionResetError): │
│ ❱ 273 │ │ │ await writer.wait_closed() │
│ 274 │ │ │ return False │
│ 275 │ else: │
│ 276 │ │ writer.writeline(line.decode(errors="replace")) │
│ │
│ /usr/local/lib/python3.12/asyncio/streams.py:364 in wait_closed │
│ │
│ 361 │ │ return self._transport.is_closing() │
│ 362 │ │
│ 363 │ async def wait_closed(self): │
│ ❱ 364 │ │ await self._protocol._get_close_waiter(self) │
│ 365 │ │
│ 366 │ def get_extra_info(self, name, default=None): │
│ 367 │ │ return self._transport.get_extra_info(name, default) │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
BrokenPipeErrorJustin Yang
05/08/2025, 7:39 PM Copy code
2025-05-08T19:38:43.737818Z [info ] Traceback (most recent call last): cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=5e9a043d-4c2e-4b6f-aaea-916e610d0501 stdio=stderr string_id=target-redshift
2025-05-08T19:38:43.738788Z [info ] File "/app/.meltano/loaders/target-redshift/venv/bin/target-redshift", line 8, in <module> cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=5e9a043d-4c2e-4b6f-aaea-916e610d0501 stdio=stderr string_id=target-redshift
2025-05-08T19:38:43.739410Z [info ] sys.exit(TargetRedshift.cli()) cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=5e9a043d-4c2e-4b6f-aaea-916e610d0501 stdio=stderr string_id=target-redshift
2025-05-08T19:38:43.739957Z [info ] ^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=5e9a043d-4c2e-4b6f-aaea-916e610d0501 stdio=stderr string_id=target-redshift
2025-05-08T19:38:43.740505Z [info ] File "/app/.meltano/loaders/target-redshift/venv/lib/python3.12/site-packages/click/core.py", line 1161, in __call__ cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=5e9a043d-4c2e-4b6f-aaea-916e610d0501 stdio=stderr string_id=target-redshift
2025-05-08T19:38:43.741037Z [info ] return self.main(*args, **kwargs) cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=5e9a043d-4c2e-4b6f-aaea-916e610d0501 stdio=stderr string_id=target-redshift
2025-05-08T19:38:43.741592Z [info ] ^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=5e9a043d-4c2e-4b6f-aaea-916e610d0501 stdio=stderr string_id=target-redshift
2025-05-08T19:38:43.742130Z [info ] File "/app/.meltano/loaders/target-redshift/venv/lib/python3.12/site-packages/click/core.py", line 1082, in main cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=5e9a043d-4c2e-4b6f-aaea-916e610d0501 stdio=stderr string_id=target-redshift
2025-05-08T19:38:43.742690Z [info ] rv = self.invoke(ctx) cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=5e9a043d-4c2e-4b6f-aaea-916e610d0501 stdio=stderr string_id=target-redshift
2025-05-08T19:38:43.743246Z [info ] ^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=5e9a043d-4c2e-4b6f-aaea-916e610d0501 stdio=stderr string_id=target-redshift
2025-05-08T19:38:43.743811Z [info ] File "/app/.meltano/loaders/target-redshift/venv/lib/python3.12/site-packages/singer_sdk/plugin_base.py", line 92, in invoke cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=5e9a043d-4c2e-4b6f-aaea-916e610d0501 stdio=stderr string_id=target-redshift
2025-05-08T19:38:43.744361Z [info ] return super().invoke(ctx) cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=5e9a043d-4c2e-4b6f-aaea-916e610d0501 stdio=stderr string_id=target-redshift
2025-05-08T19:38:43.744911Z [info ] ^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=5e9a043d-4c2e-4b6f-aaea-916e610d0501 stdio=stderr string_id=target-redshift
2025-05-08T19:38:43.745465Z [info ] File "/app/.meltano/loaders/target-redshift/venv/lib/python3.12/site-packages/click/core.py", line 1443, in invoke cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=5e9a043d-4c2e-4b6f-aaea-916e610d0501 stdio=stderr string_id=target-redshift
2025-05-08T19:38:43.746062Z [info ] return ctx.invoke(self.callback, **ctx.params) cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=5e9a043d-4c2e-4b6f-aaea-916e610d0501 stdio=stderr string_id=target-redshift
2025-05-08T19:38:43.746622Z [info ] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=5e9a043d-4c2e-4b6f-aaea-916e610d0501 stdio=stderr string_id=target-redshift
2025-05-08T19:38:43.747170Z [info ] File "/app/.meltano/loaders/target-redshift/venv/lib/python3.12/site-packages/click/core.py", line 788, in invoke cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=5e9a043d-4c2e-4b6f-aaea-916e610d0501 stdio=stderr string_id=target-redshift
2025-05-08T19:38:43.753778Z [info ] return __callback(*args, **kwargs) cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=5e9a043d-4c2e-4b6f-aaea-916e610d0501 stdio=stderr string_id=target-redshift
2025-05-08T19:38:43.754662Z [info ] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=5e9a043d-4c2e-4b6f-aaea-916e610d0501 stdio=stderr string_id=target-redshift
2025-05-08T19:38:43.755286Z [info ] File "/app/.meltano/loaders/target-redshift/venv/lib/python3.12/site-packages/singer_sdk/target_base.py", line 586, in invoke cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=5e9a043d-4c2e-4b6f-aaea-916e610d0501 stdio=stderr string_id=target-redshift
2025-05-08T19:38:43.755864Z [info ] target.listen(file_input) cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=5e9a043d-4c2e-4b6f-aaea-916e610d0501 stdio=stderr string_id=target-redshift
2025-05-08T19:38:43.756413Z [info ] File "/app/.meltano/loaders/target-redshift/venv/lib/python3.12/site-packages/singer_sdk/_singerlib/encoding/_base.py", line 53, in listen cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=5e9a043d-4c2e-4b6f-aaea-916e610d0501 stdio=stderr string_id=target-redshift
2025-05-08T19:38:43.756971Z [info ] self._process_lines(file_input or self.default_input) cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=5e9a043d-4c2e-4b6f-aaea-916e610d0501 stdio=stderr string_id=target-redshift
2025-05-08T19:38:43.757522Z [info ] File "/app/.meltano/loaders/target-redshift/venv/lib/python3.12/site-packages/singer_sdk/target_base.py", line 305, in _process_lines cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=5e9a043d-4c2e-4b6f-aaea-916e610d0501 stdio=stderr string_id=target-redshiftJustin Yang
05/08/2025, 7:40 PMoh wait did you want the original one?
Justin Yang
05/08/2025, 7:40 PM Copy code
2025-05-08T19:40:18.263914Z [info ] METRIC: {"type": "timer", "metric": "sync_duration", "value": 0.1827070713043213, "tags": {"stream": "public-amex_amexfiledownload", "pid": 857, "context": {}, "status": "succeeded"}} cmd_type=elb consumer=False job_name=scylla:tap-postgres-to-target-redshift name=tap-postgres producer=True run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=tap-postgres
2025-05-08T19:40:18.264753Z [info ] METRIC: {"type": "counter", "metric": "record_count", "value": 0, "tags": {"stream": "public-amex_amexfiledownload", "pid": 857, "context": {}}} cmd_type=elb consumer=False job_name=scylla:tap-postgres-to-target-redshift name=tap-postgres producer=True run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=tap-postgres
2025-05-08T19:40:18.265034Z [info ] Beginning full_table sync of 'public-amex_amexsponsoredfilesubmission'... cmd_type=elb consumer=False job_name=scylla:tap-postgres-to-target-redshift name=tap-postgres producer=True run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=tap-postgres
2025-05-08T19:40:18.265436Z [info ] Tap has custom mapper. Using 1 provided map(s). cmd_type=elb consumer=False job_name=scylla:tap-postgres-to-target-redshift name=tap-postgres producer=True run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=tap-postgres
2025-05-08T19:40:18.905847Z [info ] METRIC: {"type": "timer", "metric": "sync_duration", "value": 0.6418006420135498, "tags": {"stream": "public-amex_amexsponsoredfilesubmission", "pid": 857, "context": {}, "status": "succeeded"}} cmd_type=elb consumer=False job_name=scylla:tap-postgres-to-target-redshift name=tap-postgres producer=True run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=tap-postgres
2025-05-08T19:40:18.906592Z [info ] METRIC: {"type": "counter", "metric": "record_count", "value": 5028, "tags": {"stream": "public-amex_amexsponsoredfilesubmission", "pid": 857, "context": {}}} cmd_type=elb consumer=False job_name=scylla:tap-postgres-to-target-redshift name=tap-postgres producer=True run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=tap-postgres
2025-05-08T19:40:18.907181Z [info ] Beginning full_table sync of 'public-api_accesstoken'... cmd_type=elb consumer=False job_name=scylla:tap-postgres-to-target-redshift name=tap-postgres producer=True run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=tap-postgres
2025-05-08T19:40:18.907605Z [info ] Tap has custom mapper. Using 1 provided map(s). cmd_type=elb consumer=False job_name=scylla:tap-postgres-to-target-redshift name=tap-postgres producer=True run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=tap-postgres
2025-05-08T19:40:19.125805Z [info ] Found credentials from IAM Role: Wave__Data-ObservabilityScyllaGrafanaEC2Role cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.495927Z [info ] Traceback (most recent call last): cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.496400Z [info ] File "/app/.meltano/loaders/target-redshift/venv/bin/target-redshift", line 8, in <module> cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.496832Z [info ] sys.exit(TargetRedshift.cli()) cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.497205Z [info ] ^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.497645Z [info ] File "/app/.meltano/loaders/target-redshift/venv/lib/python3.12/site-packages/click/core.py", line 1161, in __call__ cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.497968Z [info ] return self.main(*args, **kwargs) cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.498385Z [info ] ^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.498693Z [info ] File "/app/.meltano/loaders/target-redshift/venv/lib/python3.12/site-packages/click/core.py", line 1082, in main cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.499045Z [info ] rv = self.invoke(ctx) cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.499394Z [info ] ^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.499745Z [info ] File "/app/.meltano/loaders/target-redshift/venv/lib/python3.12/site-packages/singer_sdk/plugin_base.py", line 92, in invoke cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.500126Z [info ] return super().invoke(ctx) cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.500483Z [info ] ^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.506078Z [info ] File "/app/.meltano/loaders/target-redshift/venv/lib/python3.12/site-packages/click/core.py", line 1443, in invoke cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.506482Z [info ] return ctx.invoke(self.callback, **ctx.params) cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.506830Z [info ] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.507165Z [info ] File "/app/.meltano/loaders/target-redshift/venv/lib/python3.12/site-packages/click/core.py", line 788, in invoke cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.507507Z [info ] return __callback(*args, **kwargs) cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.507842Z [info ] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.508186Z [info ] File "/app/.meltano/loaders/target-redshift/venv/lib/python3.12/site-packages/singer_sdk/target_base.py", line 586, in invoke cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.508531Z [info ] target.listen(file_input) cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.508869Z [info ] File "/app/.meltano/loaders/target-redshift/venv/lib/python3.12/site-packages/singer_sdk/_singerlib/encoding/_base.py", line 53, in listen cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.509207Z [info ] self._process_lines(file_input or self.default_input) cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.509567Z [info ] File "/app/.meltano/loaders/target-redshift/venv/lib/python3.12/site-packages/singer_sdk/target_base.py", line 305, in _process_lines cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.509950Z [info ] counter = super()._process_lines(file_input) cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.510265Z [info ] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.510657Z [info ] File "/app/.meltano/loaders/target-redshift/venv/lib/python3.12/site-packages/singer_sdk/_singerlib/encoding/_base.py", line 80, in _process_lines cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.510960Z [info ] self._process_schema_message(line_dict) cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshiftJustin Yang
05/08/2025, 7:40 PM Copy code
2025-05-08T19:40:19.518051Z [info ] File "/app/.meltano/loaders/target-redshift/venv/lib/python3.12/site-packages/singer_sdk/target_base.py", line 421, in _process_schema_message cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.518476Z [info ] _ = self.get_sink( cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.518831Z [info ] ^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.519170Z [info ] File "/app/.meltano/loaders/target-redshift/venv/lib/python3.12/site-packages/singer_sdk/target_base.py", line 815, in get_sink cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.519510Z [info ] return self.add_sqlsink(stream_name, schema, key_properties) cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.519851Z [info ] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.520193Z [info ] File "/app/.meltano/loaders/target-redshift/venv/lib/python3.12/site-packages/singer_sdk/target_base.py", line 749, in add_sqlsink cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.520530Z [info ] sink.setup() cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.520867Z [info ] File "/app/.meltano/loaders/target-redshift/venv/lib/python3.12/site-packages/target_redshift/sinks.py", line 82, in setup cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.521203Z [info ] with self.connector.connect_cursor() as cursor: cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.521558Z [info ] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.521955Z [info ] File "/usr/local/lib/python3.12/contextlib.py", line 137, in __enter__ cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.522257Z [info ] return next(self.gen) cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.522593Z [info ] ^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.522925Z [info ] File "/app/.meltano/loaders/target-redshift/venv/lib/python3.12/site-packages/target_redshift/connector.py", line 88, in connect_cursor cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.523262Z [info ] with redshift_connector.connect( cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.523594Z [info ] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.523929Z [info ] File "/app/.meltano/loaders/target-redshift/venv/lib/python3.12/site-packages/redshift_connector/__init__.py", line 394, in connect cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.524264Z [info ] return Connection( cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.524596Z [info ] ^^^^^^^^^^^ cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.524928Z [info ] File "/app/.meltano/loaders/target-redshift/venv/lib/python3.12/site-packages/redshift_connector/core.py", line 795, in __init__ cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.530666Z [info ] raise self.error cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshift
2025-05-08T19:40:19.531307Z [info ] redshift_connector.error.InterfaceError: {'S': 'FATAL', 'C': '28000', 'M': 'no pg_hba.conf entry for host "???", user "wave", database "scylla", SSL off', 'F': '/opt/brazil-pkg-cache/packages/RedshiftPADB/RedshiftPADB-1.0.12086.0/AL2_x86_64/generic-flavor/src/src/pg/src/backend/libpq/auth.c', 'L': '477', 'R': 'ClientAuthentication'} cmd_type=elb consumer=True job_name=scylla:tap-postgres-to-target-redshift name=target-redshift producer=False run_id=7647d3c8-301e-4458-b684-f313babc0844 stdio=stderr string_id=target-redshiftJustin Yang
05/08/2025, 7:41 PMthe last two messages are the latest stack traces
e
Edgar Ramírez (Arch.dev)
05/08/2025, 7:48 PMThanks!
Does the password contain any special characters?
j
Justin Yang
05/08/2025, 8:07 PMyes it contains
| [ + ^ ?e
Edgar Ramírez (Arch.dev)
05/08/2025, 8:42 PMHmm. That may not be it. I wonder if it's a certificate issue: https://github.com/aws/amazon-redshift-python-driver/issues/166.
j
Justin Yang
05/09/2025, 5:09 PMhey so I found the block of code:
https://github.com/TicketSwap/target-redshift/blob/main/target_redshift/connector.py#L88-L95
recreated the problem, turns out i did need ssl so I put ssl_enabled = True and the mode to ‘require’
Justin Yang
05/09/2025, 5:09 PMissue solved!
Justin Yang
05/09/2025, 5:10 PMpossibly could improve the logging here but all good
e
Edgar Ramírez (Arch.dev)
05/09/2025, 5:27 PMOh, nice!
2 Views