guisong_fu
11/21/2023, 1:37 PMLOG_BASED
set as the replication-method.
Here is some details:
First, here is the part of the meltano.yaml
file I'm using
version: 1
default_environment: prod
project_id: 0fd4eefd-b9ad-4544-8202-daac9105dcb8
environments:
- name: prod
plugins:
extractors:
- name: tap-mysql
variant: transferwise
pip_url: pipelinewise-tap-mysql
metadata:
'*':
replication-method: LOG_BASED
- name: tap-mysql--db
inherit_from: tap-mysql
config:
host: ${SOURCE_MYSQL_HOST}
port: 3306
user: ${SOURCE_MYSQL_USER}
password: ${SOURCE_MYSQL_PASSWORD}
session_sqls:
- SET @@session.net_write_timeout=3600
select:
- mock-table_d.*
- '!mock-table_d.col_1'
As you can see, I specified to exclude col_1
column from table_d
. When it runs for the first time, it does exclude col_1
column, col_1
is not there in the initial table Meltano auto created.
Then I manually added some records to table_d
with value in col_1
, and then I ran meltano run
. However, what happened now is that Meltano firstly alter table
and added col_1
, and then inserts the new the rows with value in col_1
. And that's not what I expected.
Is there anything I've done wrong?
Any suggestion would be highly appreciated!guisong_fu
11/21/2023, 1:44 PMreplication-method: LOG_BASED
. But I would be surprised that LOG_BASED
cannot be used together with column exclusion.
Any thoughts?