Hello! I'm looking for an example of how to partition a bigquery loader. Over the loader-bigquery repo I see there is a

target-tables-config.json

file but I don't know how/where to set it up. Any help is appreciated 🙂

Interesting, I wasn't aware the BQ target had partitioning support. We create the tables first, then have meltano load into it. https://github.com/adswerve/target-bigquery#setting-up-partitioning-and-clustering Not sure if meltano supports passing tap/target specific cli arguments like the

-t

option

Not sure if meltano supports passing tap/target specific cli arguments like the 

-t

  option

This is not currently supported, no. I think it would be more appropriate for the target to take this

target-tables-config

as part of its regular

--config

file. I expect that they would accept a contribution to add support for that

@great-gold-98639 when I tried to create a partitioned and then use meltano to load data, I got the error that it is not possible to merge an unpartitioned table (meltano-created) to a partitioned table (the one I created). How did you solve that?

I was wrong, @rapid-twilight-35747, seeing that error now.

@great-gold-98639 I had been thinking we could add a

tables

or

streams

dictionary right inside

config.json

, so that a separate file wouldn't be necessary anymore. Do you prefer using a separate file?

Good idea, I would rather have that as well. I just figured it's a slightly smaller change to still point to a file like the command line argument. I'll propose passing a dictionary instead.