Hi everyone. I’ve been trying for a long time to find a solution to the following problem, unfortunately no luck. I’m having problem with

select

parameters in

meltano.yml

file. First I’m not filtering out any streams/columns. All attributes are

[selected]

:

meltano select tap-redshift --list --all
Legend:
        selected
        excluded
        automatic

Enabled patterns:
        *.*

Selected attributes:
        [selected ] dev.meetings.meeting_complete_pattern_users.pattern_id
        [selected ] dev.meetings.meeting_complete_pattern_users.user_id
        [ ... meeting_complete_pattern_users - other columns ... ]
        [selected ] dev.meetings.meeting_topic_stats_aggregated.avg_negative
        [selected ] dev.meetings.meeting_topic_stats_aggregated.avg_positive
        [ ... meeting_topic_stats_aggregated - other columns ... ]

When I issue command:

meltano --environment=local-redshift elt tap-mongodb target-redshift --transform skip

data is properly copied from source to target. But then I try to select only single table. In my yaml I’ve tried (one by one) following attributes:

select:
- 'meeting_topic_stats_aggregated.*'
- 'meetings.meeting_topic_stats_aggregated.*'
- 'dev.meetings.meeting_topic_stats_aggregated.*'
- '*.meetings.meeting_topic_stats_aggregated.*'
- '*.meeting_topic_stats_aggregated.*'

but none of them works. Meaning, all streams/columns end up on the

[excluded ]

list. What I find weird is that meltano sees attributes like this:

[selected ] dev.meetings.meeting_attendees_stats.attendee

so with the

dev.

prefix, which is the name of my Redshift database. Maybe that is the problem? Following is my tap config in `meltano.yml`:

- name: tap-redshift
    variant: monad-inc
    pip_url: git+<https://github.com/Monad-Inc/tap-redshift.git>
    config:
      host: <AWS resource>
      user: <user>
      port: 5439
      dbname: dev
      schema: meetings
    capabilities:
    - catalog
    - discover
    - manifest

Any ideas why these “filters” aren’t working?

How are you running the meltano tap? Are you using

meltano run

/

meltano invoke

/

meltano elt

? I ask becuase I wonder if you're hitting https://github.com/meltano/meltano/issues/6763 An easy way to try to see if you are hitting this issue is to run

rm -rf .meltano/run

Hi Derek. I’m using

meltano elt

oh that should be working fine without any refresh, you could still try it though. (Not certain if elt stores catalog cache in /run though)

I tried removing

./tap-redshift/tap.properties.json

and trying again

glad it works, not super clear what the issue is between that and your other stuff but if it works I don't care much! If you could add what it was maybe it'd help someone 🤷

I’m still not sure why this one works and the other didn’t. Notice the difference: •

*meeting_topic_stats_aggregated.*

- works •

*.meeting_topic_stats_aggregated.*

- does not work I assumed they are the same but apparently they are not.

can you try

dev-meeting_topic_stats_aggregated.*

?

@jean_sahlberg is on point, the tap may be prefixing the stream names with the redshift schema

I’ll test it and let you know. My colleague found this issue https://gitlab.com/meltano/meltano/-/issues/2751 and that’s how he knew how to solve it