Hi there, I am running into an issue where it seems that there are temporary index tables being ingested by the

tap-sql

transferwise variant even though I have set the

catalog

config parameter to ingest a static catalog stored in the local file system that does not have these temporary index tables. I would have thought expected behavior would be to ignore these temporary index tables. The index tables seem to be dynamically and randomly generated / ingested as they don't cause the ETL run to crash on every instance. The attached log file shows that two of these temporary index tables are identified in the stream:

[2026-05-29, 13:40:07 UTC] {subprocess.py:106} INFO - 2026-05-29T13:40:07.800747Z [warning  ] Stream `64genfi6g3p5c-catalog_product_flat_1_tmp_indexer` was not found in the catalog run_id=019e73f6-aac5-777e-8dcf-0b633ea87052
[2026-05-29, 13:40:07 UTC] {subprocess.py:106} INFO - 2026-05-29T13:40:07.801008Z [warning  ] Stream `64genfi6g3p5c-catalog_product_flat_7_tmp_indexer` was not found in the catalog run_id=019e73f6-aac5-777e-8dcf-0b633ea87052

And the run subsequently crashes with the following error message:

[2026-05-29, 13:40:12 UTC] {subprocess.py:106} INFO - 2026-05-29T13:40:12.123073Z [info     ] KeyError: 'catalog_product_flat_1_tmp_indexer'

Here's the tap defined in the meltano.yml file. I have tried to specifically exclude them under the statement to no effect.

- name: tap-mysql-prod-secondary
    inherit_from: tap-mysql
    config:
      database: 64genfi6g3p5c
      port: 30001
      user: 64genfi6g3p5c
      password: xxxxxxxxxx
      catalog: extract/tap-mysql-prod.catalog.json
    metadata:
      64genfi6g3p5c-catalog_product_entity_varchar:
        replication-method: INCREMENTAL
        replication-key: value_id
        key_properties:
          - value_id
      64genfi6g3p5c-customer_entity_int:
        replication-method: INCREMENTAL
        replication-key: entity_id
        key_properties:
          - value_id
      64genfi6g3p5c-customer_entity:
        replication-method: INCREMENTAL
        replication-key: updated_at
        key_properties:
          - entity_id
      64genfi6g3p5c-customer_address_entity:
        replication-method: INCREMENTAL
        replication-key: updated_at
        key_properties:
          - entity_id
      64genfi6g3p5c-sales_order_address:
        replication-method: INCREMENTAL
        replication-key: entity_id
        key_properties:
          - entity_id
      64genfi6g3p5c-sales_order_grid:
        replication-method: INCREMENTAL
        replication-key: updated_at
        key_properties:
          - entity_id
    select:
    - 64genfi6g3p5c-blueprint_institutional_import.*
    - 64genfi6g3p5c-blueprint_sales_order_hubspot_owner.*
    - 64genfi6g3p5c-catalog_category_product.*
    - 64genfi6g3p5c-catalog_product_entity_varchar.*
    - 64genfi6g3p5c-catalog_product_entity_decimal.*
    - 64genfi6g3p5c-catalog_product_flat_1.*
    - 64genfi6g3p5c-catalog_product_flat_7.*
    - 64genfi6g3p5c-customer_address_entity.*
    - 64genfi6g3p5c-customer_entity.*
    - 64genfi6g3p5c-customer_entity_int.*
    - 64genfi6g3p5c-customer_group.*
    - 64genfi6g3p5c-eav_attribute.*
    - 64genfi6g3p5c-sales_order_address.*
    - 64genfi6g3p5c-sales_order_grid.*
    - 64genfi6g3p5c-salesrule.*
    - 64genfi6g3p5c-salesrule_coupon.*

    - "!64genfi6g3p5c-catalog_product_flat_1_tmp_indexer.*"
    - "!64genfi6g3p5c-catalog_product_flat_7_tmp_indexer.*"

The only thing I could think of would be that maybe

tap-sql

is ignoring my static config definition and still dynamically pulling in the catalog when it starts each run, thereby inadvertently pulling in these temp index tables. It seems like this issue is potentially the case as a PR was raised here, but it looks like the PR requester was ignored by the repo owner? lol 🫠 If that's the case, would I have to fork the tap and implement the change myself? Thank you for your help!

The only thing I could think of would be that maybe

tap-sql

is ignoring my static config definition and still dynamically pulling in the catalog when it starts each run, thereby inadvertently pulling in these temp index tables

I can see you have specified

catalog

under

config

, but

catalog

is actual a special type of config called an extra available to taps: https://docs.meltano.com/concepts/plugins#catalog-extra Try bumping it one level back in your configuration, in line with

config

. If that still doesn't work

would I have to fork the tap and implement the change myself?

then yes.