how to exclude table? I try use !tablename.* can't...
# troubleshooting
h
how to exclude table? I try use !tablename.* can't work
Copy code
extractors:
      - name: tap-mysql
        config:
          host: 127.0.0.1
          database: dev
          user: root
          filter_dbs: dev
          session_sqls:
          - SET @@session.net_read_timeout=36000
          - SET @@session.innodb_lock_wait_timeout=36000
        select:
        - '*.*'
        - '!dev-t_snapshot.*'
@pat_nadolny can help me?😭
p
@hello_dev if you run
meltano select tap-mysql --list
it should give you a list of streams. Did you confirm that the stream youre ignoring is in that list and is formatted the same way you listed it? Also just to confirm from our other thread, these select configs should be at the top level until the bug is fixed to respect environment level select/metadata settings
h
hi @pat_nadolny , I run
meltano select tap-mysql --list
,the output don't have dev-t_snapshot but when I run
meltano run tap-mysql target-postgres
get error about dev-t_snapshot 2022-07-08T041915.503560Z [info ] time=2022-07-08 121915 name=target_postgres level=CRITICAL message=Primary key is set to mandatory but not defined in the [dev-t_snapshot] stream cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres
p
@hello_dev thats weird 🤔. What does your properties.json look like now?
meltano invoke --dump=catalog tap-mysql > catalog.json
Have you tried just selecting 1 or 2 tables to confirm those are working? It might make it easier to understand when the issue is introduced vs using
*.*
to select everything
h
the
dev-t_snapshot
table is in
catalog.json
Copy code
select:
    - 'dev-snapshot.*'
    - '!dev-snapshot.*'
this can work the
dev-snapshot
exclude
p
@hello_dev yes it will still show up in the catalog but includes the
"selected": false,
attribute so its not synced. Wouldnt that new selection criteria exclude everything?
h
this my config ,includes the
"selected": false
,but when run also to sync
@pat_nadolny Is there something wrong with my generated config?