Hi all, I'm assessing Meltano to replicate data from 2 RDS Postgres databases into BigQuery. We'd ideally replicate data into BQ as close to realtime as possible, what's the "right" way to accomplish this using Meltano?
I was originally thinking of managing Meltano using Github actions (
like this) and setting the
cron schedule to sync every 10min or 5min, but this would have at least 2 drawbacks: 1) it would require that our meltano runs take <10min/5min and 2) it could get expensive to have github actions running non-stop all day resyncing from Postgres into BQ.
Alternatively, it seems like I could use
GCP DataStream to accomplish real time streaming into BQ, but it looks like i'd have to manage DataStream outside of Meltano.
Thanks!