Hello :clap: I am currently doing a MySQL to Click...
# troubleshooting
g
Hello 👏 I am currently doing a MySQL to Clickhouse replication using Meltano. Though I''m getting the error
ERROR    | target-clickhouse    | Record failed validation:
on a json field from a mysql table. Is there a way to load the json extracted as string into the target and not try to validate it aslo is there a way to cast the json value to string? I have tried disabling flattening and using map transformer as shown below. Any help will be appreciated My current yaml file is as below:-
Copy code
version: 1
default_environment: dev
project_id: 595874ac-c2ed-49ea-ab22-3df3483e4381
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-mysql
    variant: transferwise
    pip_url: pipelinewise-tap-mysql
    select:
    - schema-*.*
    metadata:
      schema-*:
        replication-method: LOG_BASED
  - name: tap-carbon-intensity
    variant: meltano
    pip_url: git+<https://gitlab.com/meltano/tap-carbon-intensity.git>
  loaders:
  - name: target-clickhouse
    variant: shaped-ai
    pip_url: git+<https://github.com/shaped-ai/target-clickhouse.git>
    config:
      flattening_enabled: false
  mappers:
  - name: meltano-map-transformer
    variant: meltano
    pip_url: git+<https://github.com/MeltanoLabs/meltano-map-transform.git>
    executable: meltano-map-transform
    mappings:
    - name: cast_json_fields
      config:
        flattening_enabled: False
        stream_maps:
          articles:
            images: str(images)
            videos: str(videos)
          job_adverts:
            meta: str(meta)
e
Hi @gitau_karugi! Do you still get the a validation error even after setting up the stream maps? Can you confirm that the expected fields are being stringified?
g
Hello @edgar_ramirez_mondragon Yes, I am still getting the error. I'm not sure how I can preview the stringified values. maybe you can give guidance. Thanks again
I tried as much as possible to look at the documentation. Though I feel it's lacking, mostly for the newer people starting to look at it. Some more documentation and examples I think would help. I am happy to help with this😃