does `tap csv` or `tap spreadsheets anywhere` support entity Meltano #plugins-general

does `tap-csv` or `tap-spreadsheets-anywhere` supp...

josh_lloyd

10/27/2021, 2:59 PM

does

tap-csv

tap-spreadsheets-anywhere

support entity selection?

tap-csv

doesn’t appear to do so from my testing, I get all the fields of the spreadsheet regardless of my entity selection pattern.

visch

10/27/2021, 3:20 PM

This is real new https://github.com/MeltanoLabs/tap-csv but I'd guess entity selection works based on the fact it uses the sdk!

josh_lloyd

10/27/2021, 3:37 PM

being built on the SDK apparently isn’t enough. can’t do a

meltano select --list tap-csv

on that tap or any of the others I’ve tried. I get the following error:

Copy code

Cannot list the selected attributes: Could not find catalog. Verify that the tap supports discovery mode and advertises the `discover` capability as well as either `catalog` or `properties`

josh_lloyd

10/27/2021, 3:38 PM

I’ve tried

tap-smoke-test

as well … no dice 😞

josh_lloyd

10/27/2021, 3:41 PM

to provide some clarity, I’m working on a dagster-meltano plugin for dagster and I need a way to test that the injection of environment variables is working. Using entity selection through

TAP_CSV__SELECT

seemed the easiest way to do that, but now I’m second guessing that.

visch

10/27/2021, 4:43 PM

visch@visch-ubuntu:~/git/jloyd-csvtest/csv$ meltano select --list tap-csv Legend: selected excluded automatic Enabled patterns: . Selected attributes: [automatic] test.id [selected ] test.ignore [selected ] test.name Works for me!

visch

10/27/2021, 4:43 PM

Copy code

cat test.csv
id,name,ignore
1,Josh,Secret

visch

10/27/2021, 4:43 PM

Copy code

version: 1
send_anonymous_usage_stats: true
project_id: 2b1b064b-849c-4f49-ae67-7e9590b42d19
plugins:
  extractors:
  - name: tap-csv
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/tap-csv.git>
    config:
      files:
        - entity: "test"
          path: test.csv
          keys: ["id"]

aaronsteers

10/27/2021, 4:59 PM

@josh_lloyd - Just seeing this thread. Thanks for raising. This looks like a bug and doesn't make complete sense to me:

... can’t do a
meltano select --list tap-csv
on that tap or any of the others I’ve tried. I get the following error:

Cannot list the selected attributes: Could not find catalog. Verify that the tap supports discovery mode and advertises the
discover
capability as well as either
catalog
or
properties

Probably something is failing but we're missing the failure message that actually comes from the tap. cc @florian.hines

josh_lloyd

10/27/2021, 5:00 PM

@aaronsteers thanks!

josh_lloyd

10/27/2021, 5:01 PM

@visch

*.*

works, but I want

test.id

!test.name

to work. Some sort of way of excluding one or more of the entities would prove to me that the env var injection is working

visch

10/27/2021, 5:23 PM

I thought you said

meltano select --list tap-csv

didn't work for you 😕 I have it working for me. I'd think I could filter from there

josh_lloyd

10/27/2021, 8:15 PM

@visch you’re right, that worked … I suppose my configuration was slightly different, I didn’t declare the

variant

and instead just set the

executable

and

namespace

. Not sure why that would have made a difference …

Open in Slack

Previous Next