Hello I am trying to setup a postgres to snowflake elt pipel Meltano #troubleshooting

Hello, I am trying to setup a postgres to snowfla...

aditya_penmesta

10/18/2022, 1:58 PM

Hello, I am trying to setup a postgres to snowflake elt pipeline, and for postgres I'm trying to conduct a log based replication. I have a replication slot already set up in my postgres instance, but am having trouble trying to figure out where to place this metadata in my yaml config file. Whenever I try running my pipeline, I get a

Plugin configuration is invalid HINT: WAL control functions cannot be executed during recovery.

because of this. I've been trying to comb through the documentation to figure out what exactly to do but can't seem to figure it out. Any help would be appreciated, thank you

aaronsteers

10/20/2022, 5:20 PM

Hi, @aditya_penmesta. Sorry to hear you are running into issues with the log-based replication. It looks like the Hub doesn't contain the full instructions. Can you try these instructions from Pipelinewise and let us know if this resolves your issue? https://github.com/transferwise/pipelinewise-tap-postgres#log-based-replication-requirements

aaronsteers

10/20/2022, 5:20 PM

Specifically this bit looks promising:

Copy code

SELECT *
FROM pg_create_logical_replication_slot('pipelinewise_<database_name>', 'wal2json');

If I read that correctly, the tap uses that naming convention to locate the replication slot.

aaronsteers

10/20/2022, 5:28 PM

Aiming to improve our docs: https://github.com/meltano/hub/pull/948 cc @taylor, @pat_nadolny, @Sven Balnojan

aditya_penmesta

10/21/2022, 2:11 PM

Hi AJ, I've since figured out the issue, seems like it was an issue on the Postgres end, not really the meltano side. Thank you for your help though!

aaronsteers

10/21/2022, 3:56 PM

Thanks for the update! I'm glad you got it sorted!

Open in Slack

Previous Next