Hi all. I have an issue that I'd like to see if the community can help me with. I have a base extractor which is basically tap-mysql with a bunch of configuration (selects and a bit more) on top of it. I then use this base extractor to have specific extractors per database I'm extracting from (my use case is that I have a bunch of databases all having the same tables and schemas. Now, I wanted to add a new database to the list of databases I export. I added a new extractor plugin that inherits from my base and has a filter_dbs set to the new database. However, meltano fails to see the newly added extractor and error s out with that the extractor is not known to meltano. Now, after adding an extractor manually in the yaml, what do I need to do more so that meltano also recognizes the new extractor?

meltano add --inherit-from ...

is probably a safer path. What does your

meltano.yml

look like?

Soemthing like this:

version: 1
default_environment: dev
project_id: us-cust

environments:
  - name: dev
  - name: prod

include_paths:
  - stream-mappings.yml
  - programs/*.yml

plugins:
  extractors:
    - name: source-db
      inherit_from: tap-mysql
      variant: transferwise
      pip_url: pipelinewise-tap-mysql
      config:
        host: localhost
        user: meltano
        engine: mariadb
        use_gtid: true
      select:
        - "*-Orders.*"
        - "*-Payments.*"
        - "!*-Payments.CreditCardNumber"
        - "*-Customers.*"
        - "!*-Customers.owner*"
        - "!*-Customers.secondOwner*"
        - "*-EventDataKey.*"
        - "*-Export.*"
        - "*-Note.*"
        - "*-LogItem.*"
        - "!*-LogItem.data"
      metadata:
        "*":
          replication-method: LOG_BASED

And then I have this under programs/ohio.yml (I also have similar programs/florida.yml, and so forth):

plugins:
  extractors:
  - name: parma_oh
    inherit_from: source-db
    config:
      filter_dbs: workflow_parma
  - name: linndale_oh
    inherit_from: source-db
    config
      filter_dbs: workflow_linndale
  - name: gatesmills_oh
    inherit_from: source-db
    config:
      filter_dbs: workflow_gatesmills

jobs:
  - name: oh
    tasks:
    - parma_oh snowflake
    - linndale_oh snowflake
    - gatesmills_oh snowflake
schedules:
  - name: ohio_daily
    interval: "@daily"
    job: oh

This setup has been working great for a few months, and recently I wanted to add a new city (Gates Mills, Ohio) so I went ahead and added it to the list of extractors in the ohio.yml file. However, the job fails when I run it in airflow complaining that it cannot find the extractor.

I don't see anything that jumps immediately, so I'd check for typos. Did you recently upgrade your version of Meltano?

No, I didn't. Just added gates mills and nothing else. Probably typos as you say...

I'm glad you figured it out 😁