I’m having issues with the SDK’s ability to select...
# troubleshootinga
alex_levene
05/10/2021, 5:16 PMI’m having issues with the SDK’s ability to select/de-select properties. Unsure if this is a true issue/bug or I’m just missing something. 🧵
alex_levene
05/10/2021, 5:16 PMSelect list looks like this:
Copy code
$ meltano select my-tap --list --all
Selected attributes:
[selected ] my_tap.keep_field
[excluded ] my_tap.deselect_field Copy code
$ meltano invoke --dump=catalog my-tap
...
{
"breadcrumb": [],
"metadata": {
"inclusion": "available",
"selected": true
}
}, {
"breadcrumb": [
"properties",
"keep_field"
],
"metadata": {
"inclusion": "available",
"selected": true
}
},
{
"breadcrumb": [
"properties",
"deselect_field"
],
"metadata": {
"inclusion": "available",
"selected": false
}
},
...alex_levene
05/10/2021, 5:16 PMWhen running the tap, I see unexpected events in the log and the output includes all fields (instead of selecting just
keep_field Copy code
$ meltano invoke my-tap
...
my-tap | time=2021-05-10 10:42:57 name=my-tap level=WARNING message=Catalog entry missing for 'stream_name':'('properties',)'. Using parent value of selected=True.
my-tap | time=2021-05-10 10:43:03 name=my-tap level=WARNING message=Catalog entry missing for 'stream_name':'('keep_field',)'. Using parent value of selected=True.
my-tap | time=2021-05-10 10:43:03 name=my-tap level=WARNING message=Catalog entry missing for 'stream_name':'('deselect_field',)'. Using parent value of selected=True.
...alex_levene
05/10/2021, 5:17 PMI traced this warning in the code, and noticed the breadcrumb didn’t include
properties Copy code
property_breadcrumb: Tuple[str, ...] = tuple(list(breadcrumb) + ['properties', property_name])a
aaronsteers
05/10/2021, 6:21 PM@alex_levene - Thanks for logging this. It does look like you’ve uncovered a bug. Do you feel comfortable submitting an issue and/or MR with this finding?
a
alex_levene
05/10/2021, 6:26 PMSure, no problem. Will put in the issue later today and will reference here. Thanks!
alex_levene
05/10/2021, 11:02 PMalex_levene
05/12/2021, 3:25 PM@aaronsteers I’ve put up a MR for this fix. Let me know if it looks good, thanks!
https://gitlab.com/meltano/singer-sdk/-/merge_requests/85
a
aaronsteers
05/12/2021, 3:32 PMThanks, @alex_levene. I’ll review this today after #C01QS0RV78D. Much appreciated.
a
alex_levene
05/13/2021, 7:40 PMHi @aaronsteers, do you have an idea of when you might be able to review this? Not trying to pressure, would just like to get an idea around timing. If there’s a different process for requesting review in GitLab, please let me know. Thanks in advance!