https://linen.dev logo
#plugins-general
Title
# plugins-general
j

jose

11/24/2022, 10:42 AM
Hi everyone. I've been evaluating meltano and plan to use it as part of our data stack to move data between some postgres and mysql DB into Bigquery. I'm also interested in using other taps, like jira and google ads. I've been working mainly with pipelinewise versions. The target Bigquery seems to have a behaviour that i don't see as being fit for purpose as every time it runs, tries to do a merge ( eg: It makes assumptions on the PK to use, also as the table grows this will become expensive ). Is there a way to do an append instead of a merge? I could handle the duplicates after. I've checked the adswerve version and this one seems to have a config for this, setting the
replication_method = append
What i didn't like on target-bigquery adswerve version is the way it handles the schema changes in the source db. It doesn't seem to sync that with bigquery and it fails if a new field is added in the source table.