Having a little trouble with the Meltano UI ```meltano core Meltano #troubleshooting

Having a little trouble with the Meltano UI ```mel...

david_tout

05/13/2021, 5:43 AM

Having a little trouble with the Meltano UI

Copy code

meltano.core.plugin.error.PluginNotFoundError: Transform 'tap-s3-csv' is not known to Meltano
[2021-05-13 04:57:34,081] [68404|MainThread|meltano.api] [INFO] Error: 500 Internal Server Error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.

I've got a custom extractor and transform in my project, and the UI is not happy about it....

david_tout

05/13/2021, 5:48 AM

Here's a santiised version of my

meltano.yml

file:

Copy code

version: 1
send_anonymous_usage_stats: false
project_id: <redacted>
plugins:
  extractors:
  - name: tap-s3-csv
    namespace: tap_s3_csv
    pip_url: tap-s3-csv
    executable: tap-s3-csv
    capabilities:
    - properties
    - discover
    - state
    settings:
    - name: start_date
    - name: account_id
    - name: role_name
    - name: bucket
    - name: external_id
    - name: tables
    config:
      account_id: <redacted>
      bucket: <redacted>
      role_name: <redacted>
      external_id: none
      start_date: 2020-01-01 00:00:00+00:00
      tables: <redacted>
    select:
     - ...<redacted>
    schema:
      ...<redacted>
  loaders:
  - name: target-postgres
    variant: datamill-co
    pip_url: singer-target-postgres
    config:
      logging_level: DEBUG
      disable_collection: true
      postgres_host: <redacted>
      postgres_database: postgres
      postgres_username: postgres
      postgres_schema: tap_s3_csv
      invalid_records_detect: false
      persist_empty_tables: true
  transforms:
  - name: tap-s3-csv
    pip_url: <mailto:git@gitlab.com|git@gitlab.com>:<redacted>/data/dbt-tap-s3-csv-prototype.git
    vars:
      schema: '{{ env_var(''TAP_S3_CSV__LOAD_SCHEMA'') }}'
  transformers:
  - name: dbt
    pip_url: dbt==0.18.2
  files:
  - name: dbt
    pip_url: git+<https://gitlab.com/meltano/files-dbt.git>

douwe_maan

05/13/2021, 2:38 PM

@david_tout Can you add a

namespace: tap_s3_csv

to the

name: tap-s3-csv

definition?

douwe_maan

05/13/2021, 2:39 PM

Since it's a custom plugin definition (https://meltano.com/docs/project.html#custom-plugin-definitions), rather than one shadowing a discoverable plugin (https://meltano.com/docs/project.html#shadowing-plugin-definitions)

douwe_maan

05/13/2021, 2:40 PM

And does

meltano install transform tap-s3-csv

work?

douwe_maan

05/13/2021, 2:43 PM

Note that you can also add the dbt package directly to `packages.yml`: https://meltano.slack.com/archives/C01UW21TM98/p1620315051002300?thread_ts=1620314728.001400&cid=C01UW21TM98

david_tout

05/14/2021, 1:58 AM

Thanks @douwe_maan Adding the namespace appears to have worked. btw re: DBT transform I am using Option 2. Eventually we plan to have tonnes of custom transforms (as we need to integrate different bespoke sources for different customers - hooking into ERP systems and such...) so the package approach is nice

Open in Slack

Previous Next