Meltano

Hi! I’m trying to export millions of rows from a postgres database with good performance and I was wondering if any of the posgres taps variants made use of the COPY command available to speed the export process ? I tried the datamill-co variant and it does not look like it uses it.

I don't know off the top of my head but I would check the pipelinewise variant.

Their fastsync implementation uses it, I'm sure, but I don't know if it is built directly into the plugin or if that is a fully separate codebase.

I looked a little bit in the code and it appears most of the variants use the COPY function to load CSV data into Postgres but unfortunately not in the other direction.

I did not know about fastsync, i’ll look into it but their doc states that it “is bypassing the <https://github.com/singer-io/getting-started/blob/master/docs/SPEC.md|Singer Specification> for optimised performance”. It may mean that this functionality is not available to Meltano...

<@U06CQDJRC3B> - We're building it into a Singer Spec extension so it can work in Meltano and the SDK :slightly_smiling_face::
• <https://gitlab.com/meltano/meltano/-/issues/2364|Singer extension: `BATCH` messages to enable faster throughput (#2364) · Issues · Meltano / Meltano · GitLab>
• <https://gitlab.com/meltano/sdk/-/issues/9|Add built-in support for new FAST_SYNC spec (aka BATCH message type) (#9) · Issues · Meltano / Meltano SDK for Singer Taps and Targets · GitLab>

Thanks for confirming though. I wasn't sure about the implementation of the tap behavior but it sounds like it's not yet in a place where it could be used.
<@U06CQDJRC3B> - Is this something you'd be interested in contributing for the Postgres tap? The spec extension is part of our <https://meltano.com/docs/#july-2021|roadmap this month>.

Unfortunately I don’t have the skills to contribute !

No problem. Postgres is a heavy-use tap so I do think we may get community contribution, hopefully soon after that spec goes live.