https://linen.dev logo
Join Slack
Powered by
Hello team! I'm testing the extractor `tap-clickho...
# getting-started
s
Hello team! I'm testing the extractor 
tap-clickhouse
to connect to my 
ClickHouse Cloud
instance with the following configuration:
Copy code
version: 1
default_environment: dev
project_id: 9065dc05-743d-4771-bf0b-ef744691eee1
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-clickhouse
    variant: akurdyukov
    pip_url: git+<https://github.com/akurdyukov/tap-clickhouse.git>
    config:
      username: default
      password: mypassword
      driver: http
      host: xxxxxxxxxxx.eu-central-1.aws.clickhouse.cloud
      port: 8443
      database: default
      secure: True
      verify: False
When I attempt to test the connection with:
Copy code
meltano config tap-clickhouse test
I get the following message:
Copy code
❯ meltano config tap-clickhouse test
2024-11-22T07:32:08.218900Z [info     ] The default environment 'dev' will be ignored for `meltano config`. To configure a specific environment, please use the option `--environment=<environment name>`.
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Plugin configuration is invalid
ValueError: unconverted data remains: 000
What am I doing wrong?
r
Can you run
Copy code
meltano --log-level debug config tap-clickhouse test
?
s
r
That's not the whole log, is it? I want to see from the beginning i.e. from
Copy code
2024-11-22T07:32:08.218900Z [info     ] The default environment 'dev' will be ignored for `meltano config`. To configure a specific environment, please use the option `--environment=<environment name>`.
as above.
Copy code
meltano config tap-clickhouse test &> test.out
s
r
So there is an issue parsing a date value from a response. There must be three extra `0`s than 
datetime.strptime
is expecting:
Copy code
Python 3.12.2 (main, Feb  6 2024, 20:19:44) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from datetime import datetime
>>> 
>>> x = "2024-01-01 00:00:00.000"
>>> datetime.strptime(x, '%Y-%m-%d %H:%M:%S.%f')
datetime.datetime(2024, 1, 1, 0, 0)
>>> 
>>> x = "2024-01-01 00:00:00.000000"
>>> datetime.strptime(x, '%Y-%m-%d %H:%M:%S.%f')
datetime.datetime(2024, 1, 1, 0, 0)
>>> 
>>> x = "2024-01-01 00:00:00.000000000"
>>> datetime.strptime(x, '%Y-%m-%d %H:%M:%S.%f')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/linuxbrew/.linuxbrew/Cellar/python@3.12/3.12.2_1/lib/python3.12/_strptime.py", line 554, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/linuxbrew/.linuxbrew/Cellar/python@3.12/3.12.2_1/lib/python3.12/_strptime.py", line 336, in _strptime
    raise ValueError("unconverted data remains: %s" %
ValueError: unconverted data remains: 000
>>>
👀 1
Unless you can fix the data in ClickHouse somehow (I am not familiar with it), I think there would have to be a tap fix to resolve this. There doesn't seem to be that much in the tap implementation to go wrong - in fact from the debug log, it looks like the error originates from within the ClickHouse SQLAlchemy driver.
s
it looks like the error originates from within the ClickHouse SQLAlchemy driver.
That's what I was suspecting. I was just following this https://hub.meltano.com/extractors/tap-clickhouse/ Thank you for looking into this! 👋
r
No problem, I would open an issue on the repo regardless 🙂
👍 1
7 Views
Open in Slack
PreviousNext
Powered by