Hey, I have a general question about Meltano and S...
# announcements
o
Hey, I have a general question about Meltano and Singer. We are working with few data sources that some of them are api-based and for most of them there's already a tap created in Singer, but few of them are not api based like data bases and Hadoop, and we wonder if the Singer platform together with Meltano supports such things, how common it is and is it a good practice? Or should we use directly airflow to work with this kind of sources?
d
@or_barda Using Singer taps to extract data out of databases is definitely common, and this is the use case that PipelineWise (another Singer tap runner) has focused on: https://transferwise.github.io/pipelinewise/ Their taps, e.g. https://github.com/transferwise/pipelinewise-tap-postgres, can also be used in Meltano if you add them as a custom extractor.
I can't comment on using Singer with Hadoop, but the lack of search results for "Hadoop" in the Singer Slack workspace suggests that it's not something people have tried before
o
Got it, thanks.