gitau_karugi
12/05/2023, 5:17 PMERROR | target-clickhouse | Record failed validation:
on a json field from a mysql table.
Is there a way to load the json extracted as string into the target and not try to validate it aslo is there a way to cast the json value to string?
I have tried disabling flattening and using map transformer as shown below.
Any help will be appreciated
My current yaml file is as below:-
version: 1
default_environment: dev
project_id: 595874ac-c2ed-49ea-ab22-3df3483e4381
environments:
- name: dev
- name: staging
- name: prod
plugins:
extractors:
- name: tap-mysql
variant: transferwise
pip_url: pipelinewise-tap-mysql
select:
- schema-*.*
metadata:
schema-*:
replication-method: LOG_BASED
- name: tap-carbon-intensity
variant: meltano
pip_url: git+<https://gitlab.com/meltano/tap-carbon-intensity.git>
loaders:
- name: target-clickhouse
variant: shaped-ai
pip_url: git+<https://github.com/shaped-ai/target-clickhouse.git>
config:
flattening_enabled: false
mappers:
- name: meltano-map-transformer
variant: meltano
pip_url: git+<https://github.com/MeltanoLabs/meltano-map-transform.git>
executable: meltano-map-transform
mappings:
- name: cast_json_fields
config:
flattening_enabled: False
stream_maps:
articles:
images: str(images)
videos: str(videos)
job_adverts:
meta: str(meta)
edgar_ramirez_mondragon
12/06/2023, 4:20 AMgitau_karugi
12/06/2023, 2:59 PMgitau_karugi
12/06/2023, 3:03 PM