https://linen.dev logo
Join Slack
Powered by
Hey all, I’m a newbie user of meltano. I’m trying ...
# singer-taps
w
Hey all, I’m a newbie user of meltano. I’m trying to load data from mongodb but I’m facing an issue : I have my schema definition duplicated when I invoke the tap-mongodb https://cln.sh/VKE5WZ I set the schema on meltano.yml : https://cln.sh/dFwA0k My keyword field definition is override https://cln.sh/MgEccX by the second one https://cln.sh/FqgJH7 when running postgres loader https://cln.sh/76FOSh Thanks for you support
e
Hi @walid_aziz! Do you have an example value of 
keywords
and 
tags
. I suspect the values are not actual lists, and since tap-mongodb is very "proactive" in trying to figure out a schema based on the actual record, it may be failing to even recognize these as arrays (I see 
items
is missing)
w
Hi @edgar_ramirez_mondragon here an sample : keywords”[“OK”],“tags”[“sacd”,“sacem”] I filled all my rows the same way to be sure.
But in production, I will have some rows with empty keywords tags
Is there any way to way such “proactive” behaviour ?
Hi @edgar_ramirez_mondragon, sorry to bother you but I’m really stuck. As you said, despite my schema definition, tap-mongo add a schema per row (row_to_schema) . But it does not manage to determine items type inside array.
Copy code
{'type': 'object', 'properties': {'keywords': {'anyOf': [{}]}...
Thus making target-postgres in error Could you help please. How to avoid this ?
I could comment row_to_schema feature but that would be workaround not usable on production
e
@walid_aziz other than forking the tap to make the change, perhaps trying https://github.com/transferwise/pipelinewise-tap-mongodb to see if doesn't have that weird behavior
w
I did try but same issue. I’m about forking the tap
e
fwiw this very same behavior gave me headaches in a past life: https://github.com/singer-io/tap-mongodb/issues/43
w
thanks for your inputs @edgar_ramirez_mondragon
Open in Slack
PreviousNext
Powered by