I m having some issues with using a custom loader not respec Meltano #troubleshooting

I'm having some issues with using a custom loader ...

Conner Panarella (SpaceCondor)

10/09/2024, 6:31 PM

I'm having some issues with using a custom loader not respecting the

load_schema

or other extras that should determine the schema that is used (it is using melty, since that is the default) I am trying to use my fork of

tap-postgres

(specifically on the 'use-psycopg-copy' branch). The meltano config is the following:

Copy code

extractors:

    - name: tap-shopify
      variant: matatika
      pip_url: git+<https://github.com/Matatika/tap-shopify.git@v0.3.1>
      config:
        access_token: xxxxxxx
        store: xxxxxx

Copy code

loaders:

    - name: target-postgres
      namespace: target_postgres
      pip_url: git+<https://github.com/SpaceCondor/target-postgres@use-psycopg3-copy>
      capabilities:
        - about
        - stream-maps
        - schema-flattening
        - validate-records
        - target-schema
        - hard-delete

    - name: target-postgres-dwh
      inherit_from: target-postgres
      config:
        host: xxxxxxxx
        database: xxxxxxxx
        user: meltano
        password: xxxxxxx
        add_record_metadata: true
        batch_size_rows: 100000
        validate_records: false

However, once I use my custom plugin it no longer uses

tap_shopify

as the schema, and reverts to

melty

. This is not a problem when using the meltanolabs variant (even with the same config values). Am I missing something?

✅ 1

Edgar Ramírez (Arch.dev)

10/09/2024, 9:09 PM

Hmm, let me try to reproduce that.

Edgar Ramírez (Arch.dev)

10/09/2024, 9:21 PM

I'm not able to reproduce this behavior.

meltano run tap-smoke-test target-postgres--1

uses the same schema as

meltano run tap-smoke-test target-postgres

and I see the expected tap extra:

Copy code

$ meltano --log-level warning config tap-smoke-test list --extras
_catalog [env: TAP_SMOKE_TEST__CATALOG] current value: None (default)
_state [env: TAP_SMOKE_TEST__STATE] current value: None (default)
_load_schema [env: TAP_SMOKE_TEST__LOAD_SCHEMA] current value: 'tap_smoke_test' (default: '$MELTANO_EXTRACTOR_NAMESPACE')
_select [env: TAP_SMOKE_TEST__SELECT] current value: ['*.*'] (default)
_metadata [env: TAP_SMOKE_TEST__METADATA, TAP_SMOKE_TEST_METADATA] current value: {} (default)
_schema [env: TAP_SMOKE_TEST__SCHEMA] current value: {} (default)
_select_filter [env: TAP_SMOKE_TEST__SELECT_FILTER] current value: [] (default)
_use_cached_catalog [env: TAP_SMOKE_TEST__USE_CACHED_CATALOG] current value: True (default)

meltano.yml

Edgar Ramírez (Arch.dev)

10/09/2024, 9:23 PM

both extractors have the same default value:

Edgar Ramírez (Arch.dev)

10/09/2024, 9:23 PM

Copy code

default_target_schema [env: TARGET_POSTGRES_DEFAULT_TARGET_SCHEMA] current value: None (default: '$MELTANO_EXTRACT__LOAD_SCHEMA')
default_target_schema [env: TARGET_POSTGRES__1_DEFAULT_TARGET_SCHEMA] current value: None (default: '$MELTANO_EXTRACT__LOAD_SCHEMA')

Conner Panarella (SpaceCondor)

10/10/2024, 1:05 PM

@Edgar Ramírez (Arch.dev) Could it be related to the fact that the repo was forked?

Conner Panarella (SpaceCondor)

10/10/2024, 2:14 PM

Nevermind, I was able to get it to work. My bad. I had to explicitly set default_target_schema: ${MELTANO_EXTRACT__LOAD_SCHEMA}

👍 1

8 Views

Open in Slack

Previous Next