Hi everyone. I've been evaluating meltano and plan to use it as part of our data stack to move data between some postgres and mysql DB into Bigquery. I'm also interested in using other taps, like jira and google ads. I've been working mainly with pipelinewise versions. The target Bigquery seems to have a behaviour that i don't see as being fit for purpose as every time it runs, tries to do a merge ( eg: It makes assumptions on the PK to use, also as the table grows this will become expensive ). Is there a way to do an append instead of a merge? I could handle the duplicates after. I've checked the adswerve version and this one seems to have a config for this, setting the

replication_method = append

What i didn't like on target-bigquery adswerve version is the way it handles the schema changes in the source db. It doesn't seem to sync that with bigquery and it fails if a new field is added in the source table.

I’ve been using the adswerve version for months now, to move data from mysql to bigquery. I don’t think adding columns breaks anything? Are you specifying the catalog JSON yourself?

No i'm not. I'm just adding the table i want to sync. I'm using these commands for each table

docker run  -v $(pwd):/project -w /project meltano/meltano select tap-postgres table1_report "*"

docker run  -v $(pwd):/project -w /project meltano/meltano config tap-postgres set metadata.dw-table1_report.selected true

-- select all columns

docker run  -v $(pwd):/project -w /project meltano/meltano config tap-postgres set metadata.dw-table1_report.*.selected true

Is there a better way of doing it?

Sorry I’m not used to docker commands, maybe someone else might be able to help.

Hi @abhishek_ajmera, my question was more about in how do you supply a pre-defined catalog that you want meltano to use.

There’s a

--catalog

flag that you can use for that.