If we're setting up Meltano to sync to a Bigquery ...
# announcements
a
If we're setting up Meltano to sync to a Bigquery target, anyone know if we could expect a significant performance boost or cost savings by setting up Meltano on a local GCP cloud instance, or would it be roughly equivalent to set up on an AWS instance?
d
Hi @arjun_banker! My answer here is somewhat relevant: https://meltano.slack.com/archives/CFG3C3C66/p1597679623487400?thread_ts=1597664382.487200&channel=CFG3C3C66&message_ts=1597679623.487400 I wouldn’t say either is more or less appropriate for running Meltano, but extracting and loading data is of course gonna cost you bandwidth, so I’d compare pricing on that factor before I’d look at actual machine performance, especially if GCP has a discount for “internal” traffic to BigQuery. In terms of performance, I don’t expect it to have a significant impact on how quickly your sync completes, since that’s usually limited by how quickly the extractor can extract, more than how quickly the loader can load. In general, I suggest just going with what you know!
a
That totally makes sense, in that it's constrained mostly by the underlying singer tap / sinks being utilized and the pricing/tradeoffs of the cloud service being used, will dive in that direction - thanks!
d
@arjun_banker Good luck!