Hi all, I'm assessing Meltano to replicate data fr...
# getting-started
j
Hi all, I'm assessing Meltano to replicate data from 2 RDS Postgres databases into BigQuery. We'd ideally replicate data into BQ as close to realtime as possible, what's the "right" way to accomplish this using Meltano? I was originally thinking of managing Meltano using Github actions (like this) and setting the cron schedule to sync every 10min or 5min, but this would have at least 2 drawbacks: 1) it would require that our meltano runs take <10min/5min and 2) it could get expensive to have github actions running non-stop all day resyncing from Postgres into BQ. Alternatively, it seems like I could use GCP DataStream to accomplish real time streaming into BQ, but it looks like i'd have to manage DataStream outside of Meltano. Thanks!