#8579 bug: Meltano marks block as completed and successful even if its still running or an exception has been raised. Issue created by AlejandroUPC Meltano Version 3.4.0 Python Version 3.11 Bug scope Other Operating System debian:bookworm-slim Description Context We were missing some records from our target, enabling debug logging show that the records were definitely being read but never made it into the target. We are using: •

z3-target-bigquery==0.6.8

•

meltano==3.4.0

Issue When running a LOG_BASED_REPLICATION reading

.bin

files from a MySQL tap, if a Worker is retrying any operation it will declare the run as succesfull before the workers are done. It seems to assume the process is done:

[2024-06-04T08:23:57.011+0000]  INFO - [base] 2024-06-04T08:23:56.979997Z [info     ] 2024-06-04 08:23:56,979 | INFO     | target-bigquery      | Target 'target-bigquery' completed reading 109510 lines of input (109299 records, (0 batch manifests, 130 state messages).

But shortly after adds a worker and some exception is raised (a timeout): ``` [2024-06-04T082441.764+0000] INFO - [base] 2024-06-04T082356.981168Z [info ] 2024-06-04 082356,980 | INFO | target-bigquery | Adding worker 3d1a265e893042048a55db444bfa7cf0 [2024-06-04T082441.765+0000] INFO - [base] 2024-06-04T082441.761751Z [info ] Exception in thread Thread-1: [2024-06-04T082441.766+0000] INFO - [base] 2024-06-04T082441.762195Z [info ] Traceback (most recent call last): [2024-06-04T082441.766+0000] INFO - [base] 2024-06-04T082441.762481Z [info ] File "-venv/lib/python3.11/site-packages/target_bigquery/batch_job.py", line 61, in run [2024-06-04T082441.766+0000] INFO - [base] 2024-06-04T082441.763205Z [info ] ).result() [2024-06-04T082441.767+0000] INFO - [base] 2024-06-04T082441.763517Z [info ] ^^^^^^^^ [2024-06-04T082441.767+0000] INFO - [base] 2024-06-04T082441.763690Z [info ] File "-venv/lib/python3.11/site-packages/google/cloud/bigquery/job/base.py", line 971, in result [2024-06-04T082441.768+0000] INFO - [base] 2024-06-04T082441.765473Z [info ] return super(_AsyncJob, self).result(timeout=timeout, **kwargs) [2024-06-04T082441.768+0000] INFO - [base] 2024-06-04T082441.765886Z [info ] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [2024-06-04T082441.769+0000] INFO - [base] 2024-06-04T082441.766120Z [info ] File "-venv/lib/python3.11/site-packages/google/api_core/future/polling.py", line 256, in result [2024-06-04T082441.769+0000] INFO - [base] 2024-06-04T082441.766591Z [info ] self._blocking_poll(timeout=timeout, retry=retry, polling=polling) [2024-06-04T082441.769+0000] INFO - [base] 2024-06-04T082441.766930Z [info ] File "-venv/lib/python3.11/site-packages/google/api_core/future/polling.py", line 137, in _blocking_poll [2024-06-04T082441.770+0000] INFO - [base] 2024-06-04T082441.767260Z [info ] polling(self._done_or_raise)(retry=retry) [2024-06-04T082441.770+0000] INFO - [base] 2024-06-04T082441.767538Z [info ] File "-venv/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py", line 293, in retry_wrapped_func [2024-06-04T082441.770+0000] INFO - [base] 2024-06-04T082441.768109Z [info ] return retry_target( [2024-06-04T082441.771+0000] INFO - [base] 2024-06-04T082441.768374Z [info ] ^^^^^^^^^^^^^ [2024-06-04T082441.771+0000] INFO - [base] 2024-06-04T082441.768736Z [info ] File "-venv/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py", line 153, in retry_target [2024-06-04T082441.771+0000] INFO - [base] 2024-06-04T082441.769048Z [info ] _retry_error_helper( [2024-06-04T082441.772+0000] INFO - [base] 2024-06-04T082441.769268Z [info ] File "-venv/lib/python3.11/site-packages/google/api_core/retry/retry_base.py", line 212, in _retry_error_helper [2024-06-04T082441.772+0000] INFO - [base] 2024-06-04T082441.769608Z [info ] raise final_exc from source_exc [2024-06-04T082441.772+0000] INFO - [base] 2024-06-04T082441.769975Z [info ] File "-venv/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py", line 144, in retry_target [2024-06-04T082441.772+0000] INFO - [base] 2024-06-04T082441.770223Z [info ] result = target() [2024-06-04T082441.772+0000] INFO - [base] 2024-06-04T082441.770563Z [info ] ^^^^^^^^ [2024-06-04T082441.773+0000] INFO - [base] 2024-06-04T082441.770749Z [info ] File "-venv/lib/python3.11/site-packages/google/api_core/future/polling.py", line 119, in _done_or_raise [2024-06-04T082441.773+0000] INFO - [base] 2024-06-04T082441.771086Z [info ] if not self.done(retry=retry): [2024-06-04T082441.773+0000] INFO - [base] 2024-06-04T082441.771380Z [info ] ^^^^^^^^^^^^^^^^^^^^^^ [2024-06-04T082441.774+0000] INFO - [base] 2024-06-04T082441.771664Z [info ] File "-venv/lib/python3.11/site-packages/google/cloud/bigquery/job/base.py", line 938, in done [2024-06-04T082441.774+0000] INFO - [base] 2024-06-04T082441.771900Z [info ] self.reload(retry=retry, timeout=timeout) [2024-06-04T082441.774+0000] INFO - [base] 2024-06-04T082441.772180Z [info ] File "-venv/lib/python3.11/site-packages/google/cloud/bigquery/job/base.py", line 828, in reload [2024-06-04T082441.774+0000] INFO - [base] 2024-06-04T082441.772452Z [info ] api_response = client._call_api( [2024-06-04T082441.775+0000] INFO - [base] 2024-06-04T082441.772767Z [info ] ^^^^^^^^^^^^^^^^^ [2024-06-04T082441.775+0000] INFO - [base] 2024-06-04T082441.773109Z [info ] File "-venv/lib/python3.11/site-packages/google/cloud/bigquery/client.py", line 831, in _call_api [2024-06-04T082441.775+0000] INFO - [base] 2024-06-04T082441.773515Z [info ] return call() [2024-06-04T082441.776+0000] INFO - [base] 2024-06-04T082441.773705Z [info ] ^^^^^^ [2024-06-04T082441.776+0000] INFO - [base] 2024-06-04T082441.773995Z [info ] File "-venv/lib/python3.11/site-packages/google/cloud/_http/__init__.py", line 482, in api_request [2024-06-04T082441.776+0000] INFO - [base] 2024-06-04T082441.774268Z [info ] response = self._make_request( [2024-06-04T082441.777+0000] INFO - [base] 2024-06-04T082441.774455Z [info ] ^^^^^^^^^^^^^^^^^^^ [2024-06-04T082441.777+0000] INFO - [base] 2024-06-04T082441.774677Z [info ] File "-venv/lib/python3.11/site-packages/google/cloud/_http/__init__.py", line 341, in _make_request [2024-06-04T082441.777+0000] INFO - [base] 2024-06-04T082441.774832Z [info ] return self._do_request( [2024-06-04T082441.778+0000] INFO - [base] 2024-06-04T082441.775107Z [info ] ^^^^^^^^^^^^^^^^^ [2024-06-04T082441.778+0000] INFO - [base] 2024-06-04T082441.775327Z [info ] File "-venv/lib/python3.11/site-packages/google/cloud/_http/__init__.py", line 379, in _do_request [2024-06-04T082441.778+0000] INFO - [base] 2024-06-04T082441.775659Z [info ] return self.http.request( [2024-06-04T082441.778+0000] INFO - [base] 2024-06-04T082441.775896Z [info ] ^^^^^^^^^^^^^^^^^^ [2024-06-04T082441.779+0000] INFO - [base] 2024-06-04T082441.776188Z [info ] File "-venv/lib/python3.11/site-packages/google/auth/transport/requests.py", line 541, in request [2024-06-04T082441.779+0000] INFO - [base] 2024-06-04T082441.776454Z [info ] response = super(AuthorizedSession, self).request( [2024-06-04T082441.779+0000] INFO - [base] 2024-06-04T082441.776739Z [info ] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [2024-06-04T082441.779+0000] INFO - [base] 2024-06-04T082441.777077Z [info ] File "-venv/lib/python3.11/site-packages/requests/sessions.py", line 589, in request… meltano/meltano