Hello could someone please help explain to me why this wont Meltano #troubleshooting

Hello, could someone please help explain to me why...

Bruno Arnabar

07/29/2025, 9:26 PM

Hello, could someone please help explain to me why this wont apply the mapping. I'm running:

Copy code

meltano run tap-oracle remove-clobs target-oracle

and in my yml i have:

Copy code

mappers:
  - name: meltano-map-transformer
    variant: meltano
    pip_url: meltano-map-transform
    mappings:
    - name: remove-clobs
      config:
        stream_maps:
          CUSTOMER_PHOTO:
            photo: __NULL__
            thumbnail: __NULL__
          ENVISION-CUSTOMER_PHOTO:
            photo: __NULL__
            thumbnail: __NULL__

Yet as it runs it still leads me to:

Copy code

RuntimeError: Could not convert column 'customer_photo.photo' from 'VARCHAR(4000)' to 'VARCHAR(4294967295)'

Reuben (Matatika)

07/30/2025, 1:58 AM

Casing issue maybe? Try lower-casing

CUSTOMER_PHOTO

👍 2

visch

07/30/2025, 12:24 PM

also why are you using a transformer to remove a column? You can use select for the tap just thought I'd mention it 😄

ℹ️ 1

Reuben (Matatika)

07/30/2025, 12:34 PM

Good point 😅

👍 1

Bruno Arnabar

07/30/2025, 1:01 PM

Okay I'm trying this out, I configured my tap, ran

meltano invoke tap-oracle --discover

and got back the catalog then I ran

meltano select --list --all tap-oracle

Copy code

Legend:
        selected
        excluded
        automatic
        unsupported

Enabled patterns:
        ENVISION-CUSTOMER_PHOTO.*

Selected attributes:
        [automatic  ] ENVISION-CUSTOMER_PHOTO.CUST_ID
        [selected   ] ENVISION-CUSTOMER_PHOTO.PHOTO
        [selected   ] ENVISION-CUSTOMER_PHOTO.PHOTOMODIFIEDDATE
        [selected   ] ENVISION-CUSTOMER_PHOTO.THUMBNAIL
        [selected   ] ENVISION-CUSTOMER_PHOTO.THUMBNAILMODIFIEDDATE

so then I ran

meltano select tap-oracle --rm ENVISION-CUSTOMER_PHOTO PHOTO

meltano select tap-oracle --rm "ENVISION-CUSTOMER_PHOTO" "PHOTO"

meltano select tap-oracle --rm -exclude "ENVISION-CUSTOMER_PHOTO"

"PHOTO"

Essentially trying the different approaches laid out in https://docs.meltano.com/reference/command-line-interface/#select But it seems that no pattern was changed

Reuben (Matatika)

07/30/2025, 1:11 PM

--rm

removes patterns. You want:

Copy code

meltano select tap-oracle ENVISION-CUSTOMER_PHOTO
meltano select tap-oracle --exclude ENVISION-CUSTOMER_PHOTO PHOTO

or just put

Copy code

select:
  -  'ENVISION-CUSTOMER_PHOTO.*'
  - '!ENVISION-CUSTOMER_PHOTO.PHOTO'

in your

meltano.yml

💯 1

Bruno Arnabar

07/30/2025, 1:16 PM

Okay so that did work to remove that column as you described:

Copy code

Legend:
        selected
        excluded
        automatic
        unsupported

Enabled patterns:
        ENVISION-CUSTOMER_PHOTO.*
        !ENVISION-CUSTOMER_PHOTO.PHOTO
        !ENVISION-CUSTOMER_PHOTO.THUMBNAIL

Selected attributes:
        [automatic  ] ENVISION-CUSTOMER_PHOTO.CUST_ID
        [excluded   ] ENVISION-CUSTOMER_PHOTO.PHOTO
        [selected   ] ENVISION-CUSTOMER_PHOTO.PHOTOMODIFIEDDATE
        [excluded   ] ENVISION-CUSTOMER_PHOTO.THUMBNAIL
        [selected   ] ENVISION-CUSTOMER_PHOTO.THUMBNAILMODIFIEDDATE

But when I run it still looks to resolve Photo:

Copy code

RuntimeError: Could not convert column 'customer_photo.photo' from 'VARCHAR(4000)' to 'VARCHAR(4294967295)'. cmd_type=elb consumer=True job_name=dev:tap-oracle-to-target-oracle name=target-oracle producer=False run_id=38318763-dcbd-4d24-bab2-4acc9f56329e stdio=stderr string_id=target-oracle

When its excluded from the tap does that column still load onto the target?

visch

07/30/2025, 1:17 PM

Now it's probably caching 😕 You can turn off catalog caching here https://docs.meltano.com/concepts/plugins/?meltano-tabs=terminal#use_cached_catalog-extra or remove the cache manually, whatever you want!

💡 1

visch

07/30/2025, 1:17 PM

Getting close

visch

07/30/2025, 1:18 PM

I'm almost certainly wrong after thinking about it for a second, when you add select statements the cache should automatically get invalidated

Reuben (Matatika)

07/30/2025, 1:39 PM

What does the catalog look like after your selection rules are applied?

Copy code

meltano invoke --dump catalog tap-oracle > catalog.json

Bruno Arnabar

07/30/2025, 2:04 PM

catalog.json

Reuben (Matatika)

07/30/2025, 3:42 PM

Copy code

{
          "breadcrumb": [
            "properties",
            "PHOTO"
          ],
          "metadata": {
            "sql-datatype": "BLOB",
            "inclusion": "available",
            "selected-by-default": true,
            "selected": true
          }
        },

PHOTO

is selected, despite your selection criteria... 🤔

Edgar Ramírez (Arch.dev)

07/30/2025, 6:35 PM

I don't think the tap actually honors propery-level selection metadata: https://github.com/s7clarke10/pipelinewise-tap-oracle/blob/53bb75ed27d7796d2f492e74cec87f10f1bce4d4/tap_oracle/__init__.py#L592C26-L596

🤔 1

Bruno Arnabar

07/30/2025, 6:40 PM

Oh okay, so what is the standard way to go about it in this case?

Reuben (Matatika)

07/30/2025, 8:14 PM

Given this extra context, you were probably on the right path with

meltano-map-transformer

actually. I would try the casing change now we know the properties are upper-case:

Copy code

stream_maps:
          ENVISION-CUSTOMER_PHOTO:
            PHOTO: __NULL__
            THUMBNAIL: __NULL__

👍 3

Open in Slack