https://linen.dev logo
Join Slack
Powered by
Hey all :wave: We recently got our singer tap adde...
# singer-taps
j
Hey all 👋 We recently got our singer tap added to the meltano hub but I am running into some issues running this. When I try to run this with meltano I get the following `Extractor 'tap-fleetio' is not known to Meltano. Try running 
meltano lock --update --all
to ensure your plugins are up to date.`
Interesting enough, it works fine with 
meltano add
and 
meltano install
but fails if I use 
meltano
and any other commands after to execute a pipeline
e
Hey @john_mizerany! I just tried
Copy code
meltano add extractor tap-fleetio
followed by
Copy code
meltano invoke tap-fleetio --about
and it ran successfully. I'm curious if you're running in a docker or CI environment that's missing the lock file at 
plugins/extractors/tap-fleetio--fleetio.lock
j
Yeah, I am actually running this on Prefect (subsequently K8 pods) but it does have access to that filepath / lock file
e
So how does the workflow look like there at a high level? I'm not super familiar with Prefect so I'm curious what steps/tasks are doing with the Meltano CLI.
j
Yeah! Basically we are using 
os
to run shell scripts inside our flow, we basically are just putting our environment vairables first (for the tap and target) and then afterwards just running
Copy code
meltano el tap-fleetio target-snowflake
I even tried to add in
Copy code
meltano lock --update --all
meltano el tap-fleetio target-snowflake
But I am still getting the same error
e
So when does 
meltano add
happen?
j
Sorry for just now responding, but it happens like this
Copy code
meltano add extractor tap-fleetio
meltano add loader target-snowflake
meltano lock --update --all
meltano el tap-fleetio target-snowflake
e
As part of the same Prefect pipeline?
j
Yes correct!
Hey 👋 apologies for messaging again, but this is also happening when I try to build a docker image using these docs https://docs.meltano.com/guide/containerization/
This is actually pretty interesting to find I was curious and ran this
Copy code
meltano install extractor tap-github
Extractor 'tap-fleetio' is not known to Meltano. Try running `meltano lock --update --all` to ensure your plugins are up to date.
It’s stuck on my tap and I’m not entirely sure why
e
what does your 
meltano.yml
look like?
j
Copy code
version: 1
default_environment: prod
project_id: 0e4bcc18-a3a3-4a6b-a0e3-a9ca8defbbea
environments:
- name: prod
  config:
    plugins:
      extractors:
      - name: tap-fleetio
        config:
          api_key: ${TAP_FLEETIO_API_KEY}
          account_token: $TAP_FLEETIO_ACCOUNT_TOKEN}
      loaders:
      - name: target-snowflake
        config:
          account: ${TARGET_SNOWFLAKE_ACCOUNT}
          database: ${TARGET_SNOWFLAKE_DATABASE}
          default_target_schema: ${TARGET_SNOWFLAKE_DEFAULT_TARGET_SCHEMA}
          password: ${TARGET_SNOWFLAKE_PASSWORD}
          role: ${TARGET_SNOWFLAKE_ROLE)
          schema: ${TARGET_SNOWFLAKE_SCHEMA}
          user: ${TARGET_SNOWFLAKE_USER}
          warehouse: ${TARGET_SNOWFLAKE_WAREHOUSE}
plugins:
  extractors:
  - name: tap-fleetio
    pip_url: git+<https://github.com/fleetio/tap-fleetio.git>
    config:
      api_key: ${TAP_FLEETIO_API_KEY}
      account_token: $TAP_FLEETIO_ACCOUNT_TOKEN}
  loaders:
  - name: target-snowflake
    pip_url: meltanolabs-target-snowflake
    config:
      account: ${TARGET_SNOWFLAKE_ACCOUNT}
      database: ${TARGET_SNOWFLAKE_DATABASE}
      default_target_schema: ${TARGET_SNOWFLAKE_DEFAULT_TARGET_SCHEMA}
      password: ${TARGET_SNOWFLAKE_PASSWORD}
      role: ${TARGET_SNOWFLAKE_ROLE)
      schema: ${TARGET_SNOWFLAKE_SCHEMA}
      user: ${TARGET_SNOWFLAKE_USER}
      warehouse: ${TARGET_SNOWFLAKE_WAREHOUSE}
jobs:
- name: daily-fleetio-job
  tasks:
  - tap-fleetio target-snowflake
e
and the contents of 
plugins/extractors/
?
j
Copy code
{
  "plugin_type": "extractors",
  "name": "tap-fleetio",
  "namespace": "tap_fleetio",
  "variant": "fleetio",
  "label": "Fleetio",
  "docs": "<https://hub.meltano.com/extractors/tap-fleetio--fleetio>",
  "repo": "<https://github.com/fleetio/tap-fleetio>",
  "pip_url": "git+<https://github.com/fleetio/tap-fleetio.git>",
  "executable": "tap-fleetio",
  "description": "Fleet maintenance software for fleets of all sizes",
  "logo_url": "<https://hub.meltano.com/assets/logos/extractors/fleetio.svg>",
  "capabilities": [
    "about",
    "batch",
    "catalog",
    "discover",
    "schema-flattening",
    "state",
    "stream-maps"
  ],
  "settings_group_validation": [
    [
      "account_token",
      "api_token"
    ]
  ],
  "settings": [
    {
      "name": "account_token",
      "kind": "password",
      "label": "Account Token",
      "description": "Account Token",
      "sensitive": true
    },
    {
      "name": "api_token",
      "kind": "password",
      "label": "API Token",
      "description": "The token to authenticate against the Fleetio API",
      "sensitive": true
    },
    {
      "name": "api_url",
      "kind": "string",
      "value": "<https://secure.fleetio.com/api>",
      "label": "API URL",
      "description": "Fleetio API base url"
    },
    {
      "name": "batch_config.encoding.compression",
      "kind": "options",
      "label": "Batch Config Encoding Compression",
      "description": "Compression format to use for batch files.",
      "options": [
        {
          "label": "Gzip",
          "value": "gzip"
        },
        {
          "label": "None",
          "value": "none"
        }
      ]
    },
    {
      "name": "batch_config.encoding.format",
      "kind": "options",
      "label": "Batch Config Encoding Format",
      "description": "Format to use for batch files.",
      "options": [
        {
          "label": "Jsonl",
          "value": "jsonl"
        },
        {
          "label": "Parquet",
          "value": "parquet"
        }
      ]
    },
    {
      "name": "batch_config.storage.prefix",
      "kind": "string",
      "label": "Batch Config Storage Prefix",
      "description": "Prefix to use when writing batch files."
    },
    {
      "name": "batch_config.storage.root",
      "kind": "string",
      "label": "Batch Config Storage Root",
      "description": "Root path to use when writing batch files."
    },
    {
      "name": "flattening_enabled",
      "kind": "boolean",
      "label": "Flattening Enabled",
      "description": "'True' to enable schema flattening and automatically expand nested properties."
    },
    {
      "name": "flattening_max_depth",
      "kind": "integer",
      "label": "Flattening Max Depth",
      "description": "The max depth to flatten schemas."
    },
    {
      "name": "stream_map_config",
      "kind": "object",
      "label": "Stream Map Config",
      "description": "User-defined config values to be used within map expressions."
    },
    {
      "name": "stream_maps",
      "kind": "object",
      "label": "Stream Maps",
      "description": "Config object for stream maps capability. For more information check out [Stream Maps](<https://sdk.meltano.com/en/latest/stream_maps.html>)."
    }
  ]
}
^ actually, looking at this closer this is not right. I merged a PR to change our authentication from 
api_token
-> 
api_key
. And I can still see it listed as 
api_token
But I assume that is coming from meltano hub?
e
Yeah, Meltano Hub hasn't picked up on the change it seems. Fastest way to fix that is a PR.
j
Gotcha, yeah I tried to push up my changes but I don’t have access to contribute
^ to meltano hub
e
Oh, you need to fork the repo
j
Ah 🙂 That makes sense thank you. Just out of curiosity, would this fix the issue I am seeing?
e
I'm not sure since I can't reproduce your issue by following the same steps your Prefect pipeline does
Erm.. actually I can reproduce your issue after copying your meltano.yml. One minute...
j
Thanks! For reference running 
meltano install --clean
locally also is reproducing this issue
e
Oook. For some reason your plugins in 
meltano.yml
are missing the 
variant
key:
Copy code
plugins:
  extractors:
  - name: tap-fleetio
    variant: fleetio
    pip_url: git+<https://github.com/fleetio/tap-fleetio.git>
So you probably added them manually?
j
Ahhhhhhhh yeah!
I was wondering if I needed that. I guess since this was developed by us (fleetio) and not y’all so that’s why it’s marked as the fleetio variant?
e
All plugins are expected to have a variant, since for example different implementations of target-snowflake may have different settings and such.
But I'm taking note of logging a better error message when a lock file for a different variant does exist
j
That fixed the issue!
Thank you so much, I was referring to an older meltano.yml file when we ran a poc with Meltano cloud, so I forgot to add in the variants. This is also working now with prefect
e
Amazing!
j
Apologies for all the back and forth, that is a good note to keep in mind
e
No worries, thanks for bearing with me until I found the root cause 😅
j
I did go ahead and open that PR for the hub! Thank you again for the clarification on how to contribute there
ty 1
Actually if anyone comes up with another prefect question, easiest way is to run with their 
ShellOperation
task Basically can just
Copy code
ShellOperation(
    commands = [
        "command 1",
        "command 2",
    ]
).run()
🙌 1
6 Views
Open in Slack
PreviousNext
Powered by