Hi Guys, I am testing things out I have setup clickhouse on docker ,dagster and meltano on my machine I have meltano job setup

jobs:
- name: loan_service_loader
  tasks:
  - tap-postgres target-clickhouse

when I run this i am getting error:

BrokenPipeError: [Errno 32] Broken pipe

I can insert data into clickhouse using python

Scroll up in the logs and you should find the offending stack trace with the raised exception

so if i replicate all the tables at a time then i get this error but if i replicate one table it works fine , something to do with clickhouse config ?

something to do with clickhouse config ?

Maybe. It might help us to look at the logs.

@Edgar Ramírez (Arch.dev)

2024-07-30 10:39:07 +0500 - dagster - INFO - loan_service_job - 1e32e398-386e-4257-91b9-5c77fa228886 - loan_service_loader - │ /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/streams.py:364 in      │
2024-07-30 10:39:07 +0500 - dagster - INFO - loan_service_job - 1e32e398-386e-4257-91b9-5c77fa228886 - loan_service_loader - │ wait_closed                                                                                      │
2024-07-30 10:39:07 +0500 - dagster - INFO - loan_service_job - 1e32e398-386e-4257-91b9-5c77fa228886 - loan_service_loader - │                                                                                                  │
2024-07-30 10:39:07 +0500 - dagster - INFO - loan_service_job - 1e32e398-386e-4257-91b9-5c77fa228886 - loan_service_loader - │   361 │   │   return self._transport.is_closing()                                                │
2024-07-30 10:39:07 +0500 - dagster - INFO - loan_service_job - 1e32e398-386e-4257-91b9-5c77fa228886 - loan_service_loader - │   362 │                                                                                          │
2024-07-30 10:39:07 +0500 - dagster - INFO - loan_service_job - 1e32e398-386e-4257-91b9-5c77fa228886 - loan_service_loader - │   363 │   async def wait_closed(self):                                                           │
2024-07-30 10:39:07 +0500 - dagster - INFO - loan_service_job - 1e32e398-386e-4257-91b9-5c77fa228886 - loan_service_loader - │ ❱ 364 │   │   await self._protocol._get_close_waiter(self)                                       │
2024-07-30 10:39:07 +0500 - dagster - INFO - loan_service_job - 1e32e398-386e-4257-91b9-5c77fa228886 - loan_service_loader - │   365 │                                                                                          │
2024-07-30 10:39:07 +0500 - dagster - INFO - loan_service_job - 1e32e398-386e-4257-91b9-5c77fa228886 - loan_service_loader - │   366 │   def get_extra_info(self, name, default=None):                                          │
2024-07-30 10:39:07 +0500 - dagster - INFO - loan_service_job - 1e32e398-386e-4257-91b9-5c77fa228886 - loan_service_loader - │   367 │   │   return self._transport.get_extra_info(name, default)                               │
2024-07-30 10:39:07 +0500 - dagster - INFO - loan_service_job - 1e32e398-386e-4257-91b9-5c77fa228886 - loan_service_loader - │                                                                                                  │
2024-07-30 10:39:07 +0500 - dagster - INFO - loan_service_job - 1e32e398-386e-4257-91b9-5c77fa228886 - loan_service_loader - │ /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/unix_events.py:717 in  │
2024-07-30 10:39:07 +0500 - dagster - INFO - loan_service_job - 1e32e398-386e-4257-91b9-5c77fa228886 - loan_service_loader - │ _write_ready                                                                                     │
2024-07-30 10:39:07 +0500 - dagster - INFO - loan_service_job - 1e32e398-386e-4257-91b9-5c77fa228886 - loan_service_loader - │                                                                                                  │
2024-07-30 10:39:07 +0500 - dagster - INFO - loan_service_job - 1e32e398-386e-4257-91b9-5c77fa228886 - loan_service_loader - │    714 │   │   assert self._buffer, 'Data should not be empty'                                   │
2024-07-30 10:39:07 +0500 - dagster - INFO - loan_service_job - 1e32e398-386e-4257-91b9-5c77fa228886 - loan_service_loader - │    715 │   │                                                                                     │
2024-07-30 10:39:07 +0500 - dagster - INFO - loan_service_job - 1e32e398-386e-4257-91b9-5c77fa228886 - loan_service_loader - │    716 │   │   try:                                                                              │
2024-07-30 10:39:07 +0500 - dagster - INFO - loan_service_job - 1e32e398-386e-4257-91b9-5c77fa228886 - loan_service_loader - │ ❱  717 │   │   │   n = os.write(self._fileno, self._buffer)                                      │
2024-07-30 10:39:07 +0500 - dagster - INFO - loan_service_job - 1e32e398-386e-4257-91b9-5c77fa228886 - loan_service_loader - │    718 │   │   except (BlockingIOError, InterruptedError):                                       │
2024-07-30 10:39:07 +0500 - dagster - INFO - loan_service_job - 1e32e398-386e-4257-91b9-5c77fa228886 - loan_service_loader - │    719 │   │   │   pass                                                                          │
2024-07-30 10:39:07 +0500 - dagster - INFO - loan_service_job - 1e32e398-386e-4257-91b9-5c77fa228886 - loan_service_loader - │    720 │   │   except (SystemExit, KeyboardInterrupt):                                           │
2024-07-30 10:39:07 +0500 - dagster - INFO - loan_service_job - 1e32e398-386e-4257-91b9-5c77fa228886 - loan_service_loader - ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
2024-07-30 10:39:07 +0500 - dagster - INFO - loan_service_job - 1e32e398-386e-4257-91b9-5c77fa228886 - loan_service_loader - BrokenPipeError: [Errno 32] Broken pipe

Oh, it might be a OOM issue. Does the target give you a way of reducing the record batch size?