Running into an issue with failing tasks (as reported by Airflow) that seem to result from logging:

433dfe1ab95\": 1, \"ff4f98810984614d47f6345c68d793a4\": 1, \"ff68320c47644496af1993aca13934ca\": 3, \"ff7ba98e1e4e337a8972c6f1c6f2e990\": 1, \"ff9f32b0e7c6b1153aa319f2ff9ab566\": 3, \"ffbb362fc3d0d51babc6d13b4f0a5c1e\": 3, \"ffcdeea03554eec7af5b45ffdae5f2c5\": 1, \"ffd250db4a4bb7cfda802556e0a94284\": 2, \"ffd44cbb6e259011d18316a8a3be1c60\": 3, \"ffe89a8498f31c4b365d478764fa23b0\": 3, \"ffeed351a36b77a13243280a993eefa7\": 1}", "start_timestamp": "2021-07-03T02:00:54.352318+00:00", "sync_session_id": "8f441028-5ff6-4aea-89b7-fd1b708b7cca"}, "version": 1633494224676, "time_extracted": "2021-10-11T16:45:54.889436Z"}
meltano                                 | DEBUG Deleted configuration at /project/.meltano/run/elt/kdp-el/3fd30348-fd2f-4299-b506-7c7aaf4945de/target.d8599454-0665-4781-b3a9-facde15a751a.config.json
meltano                                 | DEBUG Deleted configuration at /project/.meltano/run/elt/kdp-el/3fd30348-fd2f-4299-b506-7c7aaf4945de/tap.e0565069-91fa-4765-8c12-b9b53ebc5588.config.json
meltano                                 | time=2021-10-11 16:43:34 name=tap_postgres level=INFO message=Discovering db kolibri
meltano                                 | ERROR [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/meltano/core/logging/output_logger.py", line 135, in redirect_logging
    yield
  File "/usr/local/lib/python3.7/site-packages/meltano/cli/elt.py", line 230, in _redirect_output
    raise
  File "/usr/local/lib/python3.7/site-packages/async_generator/_util.py", line 53, in __aexit__
    await self._agen.athrow(type, value, traceback)
  File "/usr/local/lib/python3.7/site-packages/meltano/core/logging/output_logger.py", line 169, in redirect_stderr
    yield
  File "/usr/local/lib/python3.7/site-packages/async_generator/_util.py", line 53, in __aexit__
    await self._agen.athrow(type, value, traceback)
  File "/usr/local/lib/python3.7/site-packages/meltano/core/logging/output_logger.py", line 157, in writer
    await reader
  File "/usr/local/lib/python3.7/site-packages/meltano/core/logging/output_logger.py", line 205, in _read_from_fd
    await capture_subprocess_output(reader, self)
  File "/usr/local/lib/python3.7/site-packages/meltano/core/logging/utils.py", line 104, in capture_subprocess_output
    if not await _write_line_writer(writer, line):
  File "/usr/local/lib/python3.7/site-packages/meltano/core/logging/utils.py", line 81, in _write_line_writer
    writer.writeline(line.decode())
  File "/usr/local/lib/python3.7/site-packages/meltano/core/logging/output_logger.py", line 174, in writeline
    click.echo(self.prefix + line, nl=False, file=self)
  File "/usr/local/lib/python3.7/site-packages/click/utils.py", line 273, in echo
    file.flush()
  File "/usr/local/lib/python3.7/site-packages/meltano/core/logging/output_logger.py", line 184, in flush
    self.stream.flush()
BrokenPipeError: [Errno 32] Broken pipe

does the meltano job work normally outside of Airflow?

let me check.. would it be something like

meltano elt...

?

yup, exactly

the

elt

and

invoke

commands are erroring out, failing catalog discovery and I can't pass a

config.json

to either, apparently

you can pass a

config.json

to invoke:

meltano invoke tap-postgres --config config.json

Yeah, this is not proving to be at all easy. First I put together a

config.json

file, then I created a

catalog.json

file, now the tap is telling me

message=No streams marked as currently_syncing in state file