New Release of

target-bigquery

🚨 (pasted this from github in case you don't know anything about this target) • Autoscaling self-healing worker pool using either threads (default) or multiprocessing, configurable by the user for the fastest possible data ingestion. Particularly when leveraging colocated compute in GCP. • Denormalized load pattern where data is unpacked in flight into a statically typed BigQuery schema derived from the input stream json schemas. • Fix schema load pattern where all data is loaded into a

JSON

column which has been GA in BigQuery since mid 2022. • Autogenerated

VIEW

support for fixed schema load patterns which essentially overlays a statically typed schema allowing you to get the best of both worlds when using fixed schema ingestion. • JIT compilation of protobuf schemas allowing the Storage Write API to use a denormalized load pattern. • BATCH message support (naturally...) 😎 My own ramblings, Its got a pretty cool

Compressor

class that checks if

gzip

is on path and uses it to compress on the fly super fast. If gzip is not on path, it uses pythons built-in gzip module. The duck-typing interplay which makes it all seamless is cool. Just in time protobuf compilation, Json schema to VIEW DDL generation, are both interesting. Each sink is implemented in its own file. Autoscaling workers is nice since we don't know until runtime where the highest load will be. It's implemented in an interesting way too. A lot I am happy about in this one. 🤓 https://github.com/z3z1ma/target-bigquery cc: @Stéphane Burwash @dan_ladd @Sven Balnojan @visch