hello all, I was running the meltano task on airfl...
# troubleshooting
p
hello all, I was running the meltano task on airflow for a fairly large amount of time, getting quite some data. I was using a parent stream and a child stream in a single task to reduce redundant API calls. I noticed a bug:
Copy code
unhandled exception during asyncio.run() shutdown
task: <Task finished name='Task-1' coro=<elt() done, defined at /venv/lib/python3.9/site-packages/meltano/cli/elt.py:159> exception=PendingRollbackError("This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: 143")
where is this bug coming from, since the logs for long running task are weirdly handled in airlfow I can't find the exact reason around this. I do have trace from datadog, so I can go deeper if need with the trace. I want to know if this is also due to
context
dictionary getting too large or some OOM error. Any direction will be super useful
v
There should be more logs with exceptions, this doesn't have enough info. asyncio is shutting down and something's throwing an exceptionwhile that's happening