Has anyone managed to actually extract a table usi...
# troubleshootingc
chris_kings-lynne
05/14/2021, 5:06 AMHas anyone managed to actually extract a table using the new
tap-postgreschris_kings-lynne
05/14/2021, 5:06 AMI’ve been trying and trying different yaml configs:
Copy code
- name: tap-postgres--terry
inherit_from: tap-postgres
config:
filter_schemas: public
select:
- campaigns.*
- notifications.*
- vouchers.*
metadata:
'*':
replication-method: INCREMENTAL
replication-key: updated_at
'campaigns':
selected: true
'notifications':
selected: true
'vouchers':
selected: truechris_kings-lynne
05/14/2021, 5:07 AMNothing I try works. I’ve been reading the code for the tap and everything.
chris_kings-lynne
05/14/2021, 5:07 AMThis is all I see in the logs:
chris_kings-lynne
05/14/2021, 5:07 AMSelected streams: []chris_kings-lynne
05/14/2021, 5:08 AMwhat actually works in this case? meltano selectors or stream metadata?
chris_kings-lynne
05/14/2021, 6:34 AMFinally, finally worked it out:
chris_kings-lynne
05/14/2021, 6:34 AM Copy code
- name: tap-postgres--terry
inherit_from: tap-postgres
config:
filter_schemas: public
select:
- public-campaigns.*
- public-notifications.*
- public-vouchers.*'
metadata:
'*':
replication-method: INCREMENTAL
replication-key: updated_atchris_kings-lynne
05/14/2021, 6:34 AMis a working yaml configuration
v
visch
05/14/2021, 12:13 PMmeltano select --list --all tap-nameheremeltano invoke tap-name --discover > ouputd
douwe_maan
05/14/2021, 1:35 PMYeah, sounds like the issue was with the stream IDs. With database taps, they're typically prefixed with the schema as you found out.
e
edward_smith
05/14/2021, 3:10 PM@chris_kings-lynne, which one is the 'new' tap-postgres?
c
chris_kings-lynne
05/14/2021, 3:11 PMit’s the pipelinewise variant. it’s been added “natively” in the latest meltano
chris_kings-lynne
05/14/2021, 3:12 PM@derek_knox catch was the rds and the compute cluster are both hidden away in completely inaccessible secure private subnets
chris_kings-lynne
05/14/2021, 3:13 PMand the compute is serverless, hard to get in there and do that
chris_kings-lynne
05/14/2021, 3:13 PMI spent a couple of hours trying to get the multiple bastions and keys and stuff going, but then there’s no NAT so I can’t get packages, etc., etc.
chris_kings-lynne
05/14/2021, 3:14 PMI should ahve just spun up a quick rds in a public subnet, created the same schema and table and tried that from my dev machine just to get the syntax right
e
edward_smith
05/14/2021, 3:30 PMGotcha. I'm using the pipelinewise variant on some large-scale replication and it seems great so far.
c
chris_kings-lynne
05/15/2021, 5:35 AMcan you give me pro tips on using logical replication?
chris_kings-lynne
05/15/2021, 5:35 AMso far my experiment uses key based
chris_kings-lynne
05/15/2021, 5:36 AM(a) are there risks against the production database that my sre team should be aware of, how does it deal with scheme changes and what does it do with hard deletes?
d
douwe_maan
05/17/2021, 3:12 PM@chris_kings-lynne The docs in https://www.stitchdata.com/docs/replication/replication-methods/log-based-incremental will broadly apply
c
chris_kings-lynne
05/18/2021, 3:25 AMCheers, yeah working through it. The real annoyance looks like schema change handling