Hi folks, I've been working on a Clickhouse Loader...
# troubleshooting
b
Hi folks, I've been working on a Clickhouse Loader for Meltano, building on top of the existing SQL Connector support with SQLAlchemy and having a pretty good time so far, I've just got one clean-up error on the automated Tap test suite that I am looking to fix. I am hitting the following error for the
test_array_data
test-case, this appears to be an issue with how the output of the job is being flushed when reading the state. Working on addressing this right now but any hints from the crowd would be appreciated!
Copy code
../../../Library/Caches/pypoetry/virtualenvs/shaped-target-clickhouse--KUkWpar-py3.9/lib/python3.9/site-packages/singer_sdk/testing/templates.py:289: in run
    super().run(config, resource, runner)
../../../Library/Caches/pypoetry/virtualenvs/shaped-target-clickhouse--KUkWpar-py3.9/lib/python3.9/site-packages/singer_sdk/testing/templates.py:115: in run
    self.test()
../../../Library/Caches/pypoetry/virtualenvs/shaped-target-clickhouse--KUkWpar-py3.9/lib/python3.9/site-packages/singer_sdk/testing/templates.py:61: in test
    self.runner.sync_all()
../../../Library/Caches/pypoetry/virtualenvs/shaped-target-clickhouse--KUkWpar-py3.9/lib/python3.9/site-packages/singer_sdk/testing/runners.py:272: in sync_all
    self.state_messages.extend(self._clean_sync_output(self.stdout))
../../../Library/Caches/pypoetry/virtualenvs/shaped-target-clickhouse--KUkWpar-py3.9/lib/python3.9/site-packages/singer_sdk/testing/runners.py:58: in _clean_sync_output
    return [json.loads(ii) for ii in lines if ii]
../../../Library/Caches/pypoetry/virtualenvs/shaped-target-clickhouse--KUkWpar-py3.9/lib/python3.9/site-packages/singer_sdk/testing/runners.py:58: in <listcomp>
    return [json.loads(ii) for ii in lines if ii]
/opt/homebrew/Cellar/python@3.9/3.9.16/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/__init__.py:346: in loads
    return _default_decoder.decode(s)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <json.decoder.JSONDecoder object at 0x1014b5310>, s = '2023-06-12 14:57:14,662 INFO sqlalchemy.engine.Engine select version()'
_w = <built-in method match of re.Pattern object at 0x1014aa330>

    def decode(self, s, _w=WHITESPACE.match):
        """Return the Python representation of ``s`` (a ``str`` instance
        containing a JSON document).

        """
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
        end = _w(s, end).end()
        if end != len(s):
>           raise JSONDecodeError("Extra data", s, end)
E           json.decoder.JSONDecodeError: Extra data: line 1 column 5 (char 4)

/opt/homebrew/Cellar/python@3.9/3.9.16/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py:340: JSONDecodeError
Oh interesting, I just fixed this by making sure the SQLAlchemy Engine echo was turned off. I am assuming this is because it was logging to stdout and confusing Meltano when trying to deserialize the job state output.