I ve installed a custom plugin for redshift as an extractor Meltano #troubleshooting

I’ve installed a custom plugin for redshift as an ...

josh_lloyd

05/10/2021, 7:45 PM

I’ve installed a custom plugin for redshift as an extractor (pip install tap-redshift). I had to do some debugging of their code but I got it working all except for the entity selection. When I do a

meltano select tap-redshift --list --all

I’ll get a list of all the tables in the

schema

set in my config. but when I try to add model selection to the

meltano.yml

file I can’t get any of the table.columns in that list highlighted in green (‘selected’) not matter what I do. It almost feels like the

select

cli command doesn’t play nice with custom plugins. Any recommendations how how to debug?

josh_lloyd

05/10/2021, 7:48 PM

For a bit more context here is the output of my

select

command:

Copy code

% meltano select tap-redshift --list --all
Legend:
        selected
        excluded
        automatic

Enabled patterns:
        matillion.referencedata.t_ref_code_set.code_set_code

Selected attributes:
        [excluded ] matillion.referencedata.t_ref_code_set.code_set_code
        [excluded ] matillion.referencedata.t_ref_code_set.code_set_id
        [excluded ] matillion.referencedata.t_ref_code_set.code_set_name
        [excluded ] matillion.referencedata.t_ref_code_set.date_modified
        [excluded ] matillion.referencedata.t_ref_code_set.modified_by
        [excluded ] matillion.referencedata.t_ref_code_set.owner_name
...

edward_smith

05/10/2021, 8:11 PM

How are you adding the selections to meltano.yml / can you give us a snippet of the

select:

section?

josh_lloyd

05/10/2021, 8:37 PM

Well, I started off like this:

Copy code

select:
- matillion.referencedata.t_ref_code_set.*

But that didn’t work. Since my last post have had limited success with adding quotes around the line

Copy code

select:
- '*.*'

which does select everything as well as removing those lines from the yml file altogether. When I do something like:

Copy code

select:
- '*.code_set_code'

I actually get some stuff selected but it’s not precisely what I want

josh_lloyd

05/10/2021, 8:40 PM

ah, I think I might have figured it out. The thing that was confusing me was all the extra periods. I thought I might need to put a

.*

for every level in a list item, but that’s not the case. Between adding the quotes and treating the first 3 period delimited items in the item list I get what I want. For example:

Copy code

select:
- '*t_ref_code_set.*'

select only that table in the referencedata schema.

douwe_maan

05/10/2021, 8:51 PM

@josh_lloyd I'm glad you found a workaround! There's no good reason why

matillion.referencedata.t_ref_code_set.*

shouldn't also work, though. Can you please file an issue for this bug?

edward_smith

05/11/2021, 4:11 PM

@douwe_maan, is the issue here that SELECT wants {entity}.{pattern} where entity is roughly a table name and pattern is a field for databases? With multiple periods, it should be necessary to escape some periods to clarify where the entity ends / pattern begins.

douwe_maan

05/11/2021, 4:19 PM

@edward_smith Right now it's (in pseudo-regex)

{entity}.{property}(.{subproperty})+

, so multiple periods should be supported, but only if they refer to subproperties. If the entity name contains periods (which may be the case for

matillion.referencedata

?), Meltano will get confused and doesn't currently support escaping the periods

douwe_maan

05/11/2021, 4:19 PM

@josh_lloyd In the case of

matillion.referencedata.t_ref_code_set.code_set_code

, where does the entity (stream/table) name stop and the property name start?

douwe_maan

05/11/2021, 4:21 PM

https://gitlab.com/meltano/meltano/-/issues/2751 says the format is

database.schema.table.*

, so periods in stream names are indeed a thing. When I get to that issue (some time this week) I'll make note of that.

Open in Slack

Previous Next