Question on meltano with tap-salesforce: I ran the

meltano elt tap-salesforce target-jsonl

and got an error after downloading more than 800 objects. I want to retry the download now, but it is surprising that there is no state in the system database. On further investigation I found there was an exception while saving state:

ERROR exception calling callback for <Future at 0x7fb9db662110 state=finished returned NoneType>
Traceback (most recent call last):
  File "/usr/lib64/python3.7/concurrent/futures/_base.py", line 324, in _invoke_callbacks
    callback(self)
  File "/usr/lib64/python3.7/asyncio/futures.py", line 365, in _call_set_state
    dest_loop.call_soon_threadsafe(_set_state, destination, source)
  File "/usr/lib64/python3.7/asyncio/base_events.py", line 732, in call_soon_threadsafe
    self._check_closed()
  File "/usr/lib64/python3.7/asyncio/base_events.py", line 479, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
Task exception was never retrieved
future: <Task finished coro=<SingerRunner.bookmark() done, defined at /home/ec2-user/.venv/meltano/lib64/python3.7/site-packages/meltano/core/runner/singer.py:133> exception=ValueError('Separator is found, but chunk is longer than limit')>
Traceback (most recent call last):
  File "/usr/lib64/python3.7/asyncio/streams.py", line 496, in readline
    line = await self.readuntil(sep)
  File "/usr/lib64/python3.7/asyncio/streams.py", line 592, in readuntil
    'Separator is found, but chunk is longer than limit', isep)
asyncio.streams.LimitOverrunError: Separator is found, but chunk is longer than limit

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ec2-user/.venv/meltano/lib64/python3.7/site-packages/meltano/core/runner/singer.py", line 135, in bookmark
    last_state = await target_stream.readline()
  File "/usr/lib64/python3.7/asyncio/streams.py", line 505, in readline
    raise ValueError(e.args[0])
ValueError: Separator is found, but chunk is longer than limit
ELT could not complete, an error happened during the process: Subprocesses didn't exit cleanly: tap(1), target(0)

How do I retry? Obviously I do not want to re-run from the beginning.

Interesting — that error indicates that the tap is writing a state line (that the target is forwarding) of more than 64KiB in length, or that the total state emitted up to that point is over 64KiB in length (see https://stackoverflow.com/a/55458913)! That's a lot of state, but not necessarily out of character for Salesforce 😄 Can you please create an issue for that particular bug, or shall I?

@douwe_maan I checked the system database and the state payload is stored as

{"error": ""}

. Not much useful. I was wondering why there was no state information saved in between object downloads.

Ah, that's a shame. From a quick look at the code, it looks like you're right: we update the state in memory whenever a new state message comes in, but we don't commit it into the system database until the pipeline completes successfully 😞 Fortunately, this has already been reported in https://gitlab.com/meltano/meltano/-/issues/2147, which I mean to get to some time this week or next

@douwe_maan Do you have any suggestion how can I re-run for now without re-downloading all data? Shall I edit the meltano.yml and select only pending objects to download or Is there a better way?

@rahul_anand Yeah, if you know some streams were extracted and loaded in full, you can temporarily deselect those and leave only the ones you still need selected. In the future, https://gitlab.com/meltano/meltano/-/issues/2155 should help 🙂

Created the issue for state with more than 64 KB: https://gitlab.com/meltano/meltano/-/issues/2166.

@rahul_anand Great, thanks, I'll triage it!