Hey everyone, I've published a blog post about how we implemented

BATCH

support for

tap-bigquery

to significantly improve performance for large volumes of data. Give it a read if you are interested in why/how we did it! 😉 https://medium.com/@rfrankel_8960/incredible-bigquery-extract-performance-3432b9cd83fe Here is an example Meltano project that uses this tap: https://github.com/Matatika/example-ga4-export

Do you lose out on any of the data that's available via the api? Does it work for google-ads too?

> Do you lose out on any of the data that's available via the api? Not that I've seen (at least for what we are interested in). Here's the GA4 export schema: https://support.google.com/analytics/answer/7029846 > Does it work for google-ads too? I believe it is possible: https://cloud.google.com/bigquery/docs/google-ads-transfer Not exactly sure what the different between an "export" and "transfer" is, but they sound like they do similar things.

awesome @Reuben (Matatika)! for the snowflake half, was it the existing MeltanoLabs/target-snowflake? You said the BigQuery exports went to a GCS bucket, did you refactor the target to work with a storage integration vs. how it currently `put`s files to an internal snowflake stage? that's something i've been thinking about/working on for tap-mssql -> target-snowflake via S3

@visch same data as the API. There are some gotchas with the GA4 -> Big Query export. e.g. the export datasets are updated for up to 72 hours with late arriving events.

Thanks @Holly Evans!

for the snowflake half, was it the existing MeltanoLabs/target-snowflake?

Yep, since that variant supports

BATCH

messages.

You said the BigQuery exports went to a GCS bucket, did you refactor the target to work with a storage integration vs. how it currently `put`s files to an internal snowflake stage?

No, there was no change to

target-snowflake

. I guess that in theory is the next performance optimisation that could be made here... 🤔

hmm okay, i'm surprised that target-snowflake accepted batch messages with

gcs://...

manifests since i've found that

put :file_uri '@~/target-snowflake/{sync_id}'

only takes local files. when i tried batch messages with

s3://...

it errored with something like "put only accepts local files"

The GCS bucket is only used as an intermediate destination for the BigQuery export files, which are then downloaded by the tap and referenced as

file://

in the

BATCH

message. If

target-snowflake

supported GCS/other storage integrations as you were saying, then this download would not be necessary - though then the requirement of the target supporting

BATCH

messages expands to

BATCH

+ GCS storage integration support. https://docs.snowflake.com/en/user-guide/data-load-s3 https://docs.snowflake.com/en/user-guide/data-load-gcs

Ahh, makes perfect sense, thank you