Hello everyone, I'm running into a very weird prob...
# troubleshooting
g
Hello everyone, I'm running into a very weird problem. The columns that I specified to exclude will be added back in the next incremental run with
LOG_BASED
set as the replication-method. Here is some details: First, here is the part of the
meltano.yaml
file I'm using
Copy code
version: 1
default_environment: prod
project_id: 0fd4eefd-b9ad-4544-8202-daac9105dcb8
environments:
- name: prod
plugins:
  extractors:
  - name: tap-mysql
    variant: transferwise
    pip_url: pipelinewise-tap-mysql
    metadata:
      '*':
        replication-method: LOG_BASED
  - name: tap-mysql--db
    inherit_from: tap-mysql
    config:
      host: ${SOURCE_MYSQL_HOST}
      port: 3306
      user: ${SOURCE_MYSQL_USER}
      password: ${SOURCE_MYSQL_PASSWORD}
      session_sqls:
      - SET @@session.net_write_timeout=3600
    select:
    - mock-table_d.*
    - '!mock-table_d.col_1'
As you can see, I specified to exclude
col_1
column from
table_d
. When it runs for the first time, it does exclude
col_1
column,
col_1
is not there in the initial table Meltano auto created. Then I manually added some records to
table_d
with value in
col_1
, and then I ran
meltano run
. However, what happened now is that Meltano firstly
alter table
and added
col_1
, and then inserts the new the rows with value in
col_1
. And that's not what I expected. Is there anything I've done wrong? Any suggestion would be highly appreciated!
I'm suspecting this is related to
replication-method: LOG_BASED
. But I would be surprised that
LOG_BASED
cannot be used together with column exclusion. Any thoughts?