https://linen.dev logo
Join Slack
Powered by
I've just noticed that fields that are not selecte...
# troubleshooting
e
I've just noticed that fields that are not selected are being replicated. Here is a snippet from meltano.yml showing only a few selected fields from the table, but the result is that all fields are being replicated:
Here you can see that meltano reports them as 
excluded
d
If Meltano shows them as excluded, they are getting marked as 
selected: true
in the catalog that is sent to tap-postgres, but it’s possible that the tap isn’t actually respecting that
e
Must be the case... this is Wise's tap postgres, here is the catalog dump:
Copy code
{
          "breadcrumb": [
            "properties",
            "password"
          ],
          "metadata": {
            "sql-datatype": "character varying",
            "inclusion": "available",
            "selected-by-default": true,
            "selected": false
          }
        },
We are trying to avoid replicating the password column among others.
d
Are you using log-based replication? It may be the same as this issue: https://github.com/transferwise/pipelinewise-tap-mysql/issues/50
e
No, we can't do anything log-based here, so this is incremental...I'm reading through the code now to see what I can figure.
Still digging on this, so far I've determined that the 
selected
field isn't being kept all the way to when the decision gets made... for example, here is a snippit from the 
md_map
while the tap is starting:
Copy code
"(""properties","first_name"")":{
      "sql-datatype":"character varying",
      "inclusion":"available",
      "selected-by-default":true
   },
@douwe_maan Thoughts on this? Here is what I get when I do 
meltano elt cb_core-4 target-snowflake --dump catalog
.. But I modded 
should_sync_column
as pictured and I don't see the selected attribute included there.
d
@edward_smith Did you trace the catalog all the way through the tap? Does 
selected: false
go missing right away? 
--dump=catalog
dumps the file exactly as it is passed to the tap, so this feels like a tap bug
e
Yes, the screenshot above is from --dump catalog
Oh, wait, on re-read I understand you to mean did I trace it all the way through. I'm not sure what the intermediate steps are, I'm trying to see where it parses the catalog and see if it sees it there
d
For those following along: Edward has found something and shared it in Singer Slack: https://singer-io.slack.com/archives/CCD7A22T0/p1624912354005600?thread_ts=1624912298.005500&cid=CCD7A22T0
e
Ok, I was able to confirm that making 
refresh_streams_schema()
a no-op fixes the issue. My reading of the code is that it assumes that the only catalog metadata that needs to be preserved from the catalog is table-level metadata.
So, it makes a copy of the table-level metadata and updates the discovered catalog with it and then uses the result. The outcome is that field-level metadata is lost
d
Does it have logic that makes it seem like it would actually respect 
selected: false
on individual properties, or is that not implemented at all?
e
It doesn't retain field-level metadata from the catalog, it replaces it with what it discovers.
But, when I put 
return
at the start of the function, my meltano select's are respected!
d
What I’m wondering is whether making it retain the field-level metadata would immediately make it work as expected, or whether that field-level metadata is never read to begin with
Aah, perfect!
e
I think it would make it work. I'll work on a patch for that tomorrow
What I've done now will fail to 'rediscover' things from the schema... I'm not sure if that is important, but I imagine it is for some other use cases
d
Yeah, I’d imagine so
e
For those following along, the code is here. This loop merges the table-level metadata between discovery and the provided catalog, but it replaces field-level metadata from the catalog with what it found in discovery: https://github.com/transferwise/pipelinewise-tap-postgres/blob/master/tap_postgres/stream_utils.py#L89
I've put a PR together for this, please provide feedback if you are able: https://github.com/transferwise/pipelinewise-tap-postgres/pull/102
2 Views
Open in Slack
PreviousNext
Powered by