I m trying to make meltano save the tables to <https hub mel Meltano #singer-targets

I'm trying to make meltano save the tables to <tar...

David Dobrinskiy

07/18/2025, 3:07 PM

I'm trying to make meltano save the tables to target-clickhouse using a template, but can't find a working example anywhere. I have tap-postgres as source, and target-clickhouse as my target. I am replicating multiple schemas from postgres, and for each

schema.table_name

I want to save them to the same database in clickhouse like this:

Copy code

raw_data.{schema}__{table_name_1}
raw_data.{schema}__{table_name_2}
...

But so far I'm confused how to edit my

meltano.yml

in such a way as to extract the schema/table_name from source and reuse it for target destination

David Dobrinskiy

07/18/2025, 3:14 PM

meltano.yml:

Copy code

version: 1
default_environment: prod
project_id: [REDACTED]
environments:
- name: prod

plugins:
  extractors:
  - name: tap-postgres 
    variant: meltanolabs
    pip_url: meltanolabs-tap-postgres
    config:
      host: ${POSTGRES_HOST}
      port: 5432
      user: ${POSTGRES_USER}
      password: ${POSTGRES_PASSWORD}
      database: prod
      # custom WHERE clauses for specific streams - fast iterations with recent data only
      stream_options:
        users:
          custom_where_clauses:
            - "next_purchase_id >= 2" # only users with at least 1 purchase
            - "updated_at >= '2025-07-15'" # only recent data for fast iterations
    metadata:
      users:
        replication-method: INCREMENTAL
        replication-key: updated_at
        primary-key: id
    select:
    - schema1-users.*
    - schema2-orders.*

  loaders:
  - name: target-clickhouse
    variant: shaped-ai
    pip_url: git+<https://github.com/shaped-ai/target-clickhouse.git>
    config:
      host: ${CH_DWH_HOST}
      port: 8123 # http port
      username: ${CH_DWH_USER}
      password: ${CH_DWH_PASSWORD}
      # THIS IS IMPORTANT: default_target_schema overwrites the destination database name in the target-clickhouse config
      # otherwise, the destination database name will be the same as the source schema name from the tap-custodial-battery config
      default_target_schema: raw
      optimize_after: true
      # WORK IN PROGRESS: THIS IS PROBABLY WRONG
      stream_maps:
        "*":
          __alias__: "'raw__' + $MELTANO_LOADER_NAME"

jobs:
- name: job-replicate
  tasks:
  - tap-postgres target-clickhouse

Open in Slack