#7333 bug: BrokenPipe Error Issue created by dimoschi Meltano Version 2.15.3 Python Version 3.9 Bug scope API Operating System Linux - Ubuntu 20.04 Description Meltano unexpectedly raises a BrokenPipe error. Meltano runs as a Prefect 1.4.1 task executed inside a k8s pod that uses a custom image built following the documentation. Code [2023-02-17 102245+0000] INFO - prefect.CloudTaskRunner | Task 'ShellTask': Starting task run... [2023-02-17 105240+0000] INFO - prefect.ShellTask | 2023-02-17T105240.324374Z [error ] [2023-02-17 105240+0000] INFO - prefect.ShellTask | ╭───────────────────── Traceback (most recent call last) ──────────────────────╮ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ /usr/local/lib/python3.9/site-packages/meltano/core/logging/output_logger.py │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ :203 in redirect_logging │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 200 │ │ │ *ignore_errors, │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 201 │ │ ) │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 202 │ │ try: │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ ❱ 203 │ │ │ yield │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 204 │ │ except ignored_errors: # noqa: WPS329 │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 205 │ │ │ raise │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 206 │ │ except RunnerError as err: │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ /usr/local/lib/python3.9/site-packages/meltano/core/block/extract_load.py:46 │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 1 in run │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 458 │ │ │ # TODO: legacy

meltano elt

style logging should be depre │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 459 │ │ │ legacy_log_handler = self.output_logger.out("meltano", log │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 460 │ │ │ with legacy_log_handler.redirect_logging(): │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ ❱ 461 │ │ │ │ await self.run_with_job() │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 462 │ │ │ │ return │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 463 │ │ else: │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 464 │ │ │ logger.warning( │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ /usr/local/lib/python3.9/site-packages/meltano/core/block/extract_load.py:48 │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 7 in run_with_job │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 484 │ │ │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 485 │ │ with closing(self.context.session) as session: │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 486 │ │ │ async with job.run(session): │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ ❱ 487 │ │ │ │ await self.execute() │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 488 │ │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 489 │ async def terminate(self, graceful: bool = False) -> None: │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 490 │ │ """Terminate an in flight ExtractLoad execution, potentially d │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ /usr/local/lib/python3.9/site-packages/meltano/core/block/extract_load.py:45 │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 3 in execute │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 450 │ │ async with self._start_blocks(): │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 451 │ │ │ await self._link_io() │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 452 │ │ │ manager = ELBExecutionManager(self) │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ ❱ 453 │ │ │ await manager.run() │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 454 │ │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 455 │ async def run(self) -> None: │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 456 │ │ """Run the ELT task.""" │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ /usr/local/lib/python3.9/site-packages/meltano/core/block/extract_load.py:65 │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ /usr/local/lib/python3.9/site-packages/meltano/core/block/extract_load.py:65 │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 0 in run │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 647 │ │ stopping blocks or waiting for IO to complete as appropriate. │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 648 │ │ the blocks exit with a non 0 exit code. │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 649 │ │ """ │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ ❱ 650 │ │ await self._wait_for_process_completion(self.elb.head) │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 651 │ │ _check_exit_codes( │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 652 │ │ │ self._producer_code, self._consumer_code, self._intermedia │ [2023-02-17 105240+0000] INFO - prefect.ShellTask | │ 653 │ │ )… meltano/meltano