Hi everyone, I banged my head against every combin...
# plugins-generala
andrew_stewart
07/17/2021, 5:40 PMHi everyone, I banged my head against every combination of mongodb tap and postgres target so that you don’t have to 😄. Here’s the verdict:
• https://github.com/transferwise/pipelinewise-tap-mongodb
• https://github.com/transferwise/pipelinewise-target-postgres
This combo seems to be the most robust to all the wackiness involved with extracting from arbitrary nosql schemas and figuring out all the various postgres targets out there in the wild.
t
taylor
07/19/2021, 2:29 PMWe can look into updating the discoverable variant within Meltano - we list Singer currently.
a
andrew_stewart
07/19/2021, 4:42 PMMeltano Hub obviously needs reddit style upvoting (jk)
andrew_stewart
07/19/2021, 4:42 PM😀
andrew_stewart
07/19/2021, 4:47 PMI think there’s a pretty interesting lesson from investigating all those combinations btw. Loosely structured data sources like Mongo (and probably any other NoSQL db) are understandably a challenge for the expectations of targets. Part of the reason that pair of pipelinewise tap/targets works well is because they don’t even bother with trying to flatten/expand and autodetect the variably nested document structures.. instead you just get simple source tables with singer metadata columns and a single
documente
edward_ryan
07/26/2021, 1:26 PM@tom_mcgrail @gunnar @monika_rajput
y
yingqiu_lee
07/30/2021, 6:15 AMPart of the reason that pair of pipelinewise tap/targets works well is because they don’t even bother with trying to flatten/expand and autodetect the variably nested document structures..I just started experimenting with Meltano as a potential ELT tool, so still trying to wrap my head around stuff. Do you know if there are pairs that flatten the source NoSQL document to flat tables?
a
andrew_stewart
07/30/2021, 9:30 PMYes, there are, but from my experience you are asking for trouble that way since it is difficult for a mongodb tap and any particular target to be able to systematically anticipate the amorphous structures being extracted and flattened (and type inferred without ambiguity).
I would strongly suggest treating any flattening operations at the transformation phase rather than extraction and loading.