Hi all, currently, my team want to use the

flattening

feature so I have a small demo to adopt this feature but it does not go as expected. I use: •

tap-postgres

as an extractor (pip_url: git+https://github.com/transferwise/pipelinewise-tap-postgres.git) •

meltano-map-transformer

as the mapper (pip_url: git+https://github.com/MeltanoLabs/meltano-map-transform.git@v0.7.0) •

target-sqlite

as the loader (pip_url: git+https://github.com/MeltanoLabs/target-sqlite.git) This is what the meltano.yml looks like:

version: 1
default_environment: dev
send_anonymous_usage_stats: false
project_id: 33080f0b-2d0f-4fba-b7f1-727bbf080f91

plugins:
  extractors:
  - name: tap-postgres
    variant: transferwise
    pip_url: git+<https://github.com/transferwise/pipelinewise-tap-postgres.git>
    metadata:
      '*':
        replication-method: FULL_TABLE
  mappers:
  - name: meltano-map-transformer
    namespace: meltano_map_transformer
    pip_url: -e .
    executable: meltano-map-transform
    settings:
    - name: stream_maps
      kind: object
    mappings:
    - name: flatten
      config:
        stream_maps: {}
        flattening_enabled: True
        flattening_max_depth: 2
  loaders:
  - name: target-sqlite
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/target-sqlite.git>
    config:
      batch_size: 200
      database: output/$MELTANO_EXTRACTOR_NAMESPACE.db

environments:
- name: dev
  config:
    plugins:
      extractors:
      - name: tap-postgres
        config:
          host: localhost
          port: 32768
          user: postgres
          password: "mysecretpassword"
          dbname: test

In the postgres database, I have the table like below. However, when i run something like

meltano run tap-postgres flatten target-sqlite

, it does not flatten the json field as expected. The schema of the replicated data in the sqlite db is like below. I'm pretty new to Meltano so this might sound stupid to you guys. If it is, please send me some documentation for me to read more about this. Thank you a lot in advance melty bouncy

You might need to tell the tap about the structure of the json field. You can do that using the schema extra:

plugins:
  extractors:
  - name: tap-postgres
    variant: transferwise
    pip_url: git+<https://github.com/transferwise/pipelinewise-tap-postgres.git>
    metadata:
      '*':
        replication-method: FULL_TABLE
    schema:
      public-test_db_jsonb:
        type: ["object", "null"]
        properties:
          id:
            type: "integer"
          val:
            type: "string"

@Edgar Ramírez (Arch.dev), I will try this. Thank you a lot, sir dancingpenguin