Has anyone attempted to use the Clickhouse Airbyte...
# getting-startedb
ben_theunissen
03/29/2023, 7:34 PMHas anyone attempted to use the Clickhouse Airbyte loader with Meltano?
t
taylor
03/29/2023, 7:50 PMLoaders aren’t currently supported by the airbyte integration - just the extractors
b
ben_theunissen
03/29/2023, 7:50 PMAh my mistake, makes sense!
ben_theunissen
03/29/2023, 7:51 PMClickhouse is something we've just landed on in our team as our likely candidate for data warehouse analytics, so I may be looking at writing a loader at some point in the near future (just don't hold me to it soon!)
t
taylor
03/29/2023, 7:51 PMAll good! I’ll go open an issue on that right now though since I don’t think there’s currently one. In theory it wouldn’t be too hard to do.
taylor
03/29/2023, 7:51 PMAh cool! well let us know if you need help on the SDK side
b
ben_theunissen
03/29/2023, 7:52 PMWould the issue be for Airbyte Extractors in general, or a Meltano SDK Clickhouse loader?
t
taylor
03/29/2023, 7:55 PMhttps://github.com/meltano/meltano/issues/7460 support for airbyte loaders
j
jan_soubusta
03/30/2023, 2:17 PMWe just implemented an in-house loader.
Most painful was that Clickhouse does not support schemas,
Looks like dbt maps CLickhouse databases to schemas, but not sure if it is possible to transform source table from schema1 to target table in schema2 (cross-database SQL).
Tough most likely it is not relevant for Meltano (single target schema).
Anyway, the Airbyte loader officially supported by Clickhouse is the best option.
b
ben_theunissen
03/30/2023, 4:08 PM@jan_soubusta Is your in-house loader built using the Meltano SDK, I would be really interested if you could share at some point in the future 🙂
j
jan_soubusta
03/30/2023, 4:12 PMUnfortunately it is not, it was built my someone else on top of our in-house "SDK".
The only thing I can share is how we manage load-related operations (open connection, (re)create schema(database), executing DDL, load files.