https://linen.dev logo
Join Slack
Powered by
Hello, I am new to Meltano and am very pleased to ...
# announcements
a
Hello, I am new to Meltano and am very pleased to be here. I am hoping to find an example of performing a data ingest from a PostGreSQL database (and some other sources, including, for example, a number of CSV files) into a set of tables in RedShift, from which reports will be generated. Thank you for having me.
j
Cc @yannis_roussos
a
I found the tap-CSV, but not seeing a tap-postgres — only in Singer, but not sure if this is the one to use, since it does not appear in the Meltano documentation. Also, would target-postgres work with RedShift?
j
We have tap postgres
a
Thank you, could you please share the link to tap postgres, @jacob_schatz ? Thanks!
j
Oh. You know what. We have target postgres. Not tap postgres.
We still would have to create that.
I'll talk to the team and see what the feasibility of that is
m
@alex_sherstinsky I know that @rob_moore have been working on integrating tap-postgres as a custom tap, maybe he could help you. As far as 
target-postgres
I don't know if it works with Redshift, but IIRC there is a target-redshift that could be added. Can you please open an issue with your use-case and we'll look into prioritizing these components?
r
@alex_sherstinsky Happy to help with 
tap-postgres
. There are some issues with it currently if you are using the incremental setting it only receives full updates. Also, I'd advise using the unreleased version (that is, pointing at github rather than using the version in Pypi).
a
@rob_moore Thanks — is the unreleased version more full-featured? If so, could you please give me the link to it? Any documentation to better understand “incremental” versus non-incremental? Thanks so much!
@micael_bergeron Has this target-redshift been shown to work already? Or is it brand new? 🙂 And what is needed to make it work under Meltano? Any special guide for including a target or tap that does not currently appear in the Meltano documentation? /cc @jacob_schatz
r
Part of the joy of using 
tap-postgres
is that you also need to use 
tap-mysql
for documentation.
Since the 
tap-postgres
docs are nearly non-existent.
The repo has a fix that I made that allows it to work better with meltano. In short, you don't have to use 
meltano invoke
with it.
You'll need to run 
meltano add --custom-extractor tap-postgres
as detailed here: https://meltano.com/docs/tutorial.html#add-the-plugin-to-your-meltano-project-custom
In terms of the replication strategy, for now you should probably just stick with 
FULL_TABLE
as 
INCREMENTAL
doesn't really work at the moment.
Some detail about the strategies, here: https://github.com/singer-io/tap-mysql#incremental
Here's the issue regarding incremental strategy: https://gitlab.com/meltano/meltano/issues/601
You might find this thread helpful (or possibly just confusing): https://meltano.slack.com/conversation/CFFBG6DS4/p1554606778003100
If you aren't looking to do anything more than the 
el
of 
elt
, then you might take a look at 
singer-runner
as its learning curve is less steep: https://github.com/datamill-co/singer-runner
a
Thank you, @rob_moore, for all this amazing and very helpful information. I will now be processing it. Thanks so much! 🙂
m
@rob_moore 🎩
8 Views
Open in Slack
PreviousNext
Powered by