Hello, I run my meltano project but unfortunately ...
# troubleshooting
e
Hello, I run my meltano project but unfortunately table i want to get is skipped. my project:
Copy code
version: 1
default_environment: dev
project_id: 6d246080-d283-4f3b-afe1-52264750c7c2
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-postgres
    namespace: tap_postgres
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/tap-postgres.git>
    config:
      port: 5432
      user: my_user
      password: my_password
      database: my_db
      replication_method: LOG_BASED
      replication_slot: my_replication_slot
      start_lsn: 0/0
      tables:
      - table: my_db.my_table
      default_replication_method: LOG_BASED
      host: my_host
  loaders:
  - name: target-clickhouse
    variant: shaped-ai
    namespace: target_clickhouse
    pip_url: git+<https://github.com/shaped-ai/target-clickhouse.git>
    config:
      host: "my_host"
      port: 8123
      user: "my_user"
      password: "my_password"
      database: "my_db"
      tables:
        - table: "my_table"
Log I got:
Copy code
meltano run tap-postgres target-clickhouse
2024-09-12T22:11:21.577887Z [debug    ] meltano 3.5.1, Linux
2024-09-12T22:11:21.585315Z [debug    ] /etc/timezone found, contents:
 Etc/UTC

2024-09-12T22:11:21.586160Z [debug    ] /etc/localtime found
2024-09-12T22:11:21.587621Z [debug    ] 2 found:
 {'/etc/timezone': 'Etc/UTC', '/etc/localtime is a symlink to': 'Etc/UTC'}
2024-09-12T22:11:21.597078Z [info     ] Environment 'dev' is active
2024-09-12T22:11:21.625345Z [debug    ] Creating DB engine for project at 'my_dir' with DB URI 'sqlite:my_dir'
2024-09-12T22:11:21.707162Z [debug    ] found plugin in cli invocation plugin_name=tap-postgres
2024-09-12T22:11:21.708571Z [debug    ] found plugin in cli invocation plugin_name=target-clickhouse
2024-09-12T22:11:21.709248Z [debug    ] head of set is extractor as expected block=tap-postgres
2024-09-12T22:11:21.905256Z [debug    ] found block                    block_type=loaders index=1
2024-09-12T22:11:21.905640Z [debug    ] blocks                         idx=1 offset=0
2024-09-12T22:11:21.971992Z [debug    ] found ExtractLoadBlocks set    offset=0
2024-09-12T22:11:21.972442Z [debug    ] All ExtractLoadBlocks validated, starting execution.
2024-09-12T22:11:22.130248Z [debug    ] Skipped installing extractor 'tap-postgres'
2024-09-12T22:11:22.194198Z [debug    ] Skipped installing loader 'target-clickhouse'
2024-09-12T22:11:22.194802Z [debug    ] Installed 0/2 plugins
2024-09-12T22:11:22.195092Z [debug    ] Skipped installing 2/2 plugins
2024-09-12T22:11:22.484866Z [debug    ] Created configuration at my_dir/.meltano/run/tap-postgres/tap.50ce392b-f1d4-4cdd-ac72-259851cd18b5.config.json
2024-09-12T22:11:22.485981Z [debug    ] Could not find tap.properties.json in my_dir/.meltano/extractors/tap-postgres/tap.properties.json, skipping.
2024-09-12T22:11:22.486638Z [debug    ] Could not find tap.properties.cache_key in my_dir/.meltano/extractors/tap-postgres/tap.properties.cache_key, skipping.
2024-09-12T22:11:22.487188Z [debug    ] Could not find state.json in my_dir/.meltano/extractors/tap-postgres/state.json, skipping.
2024-09-12T22:11:22.493536Z [debug    ] Invoking: ['my_dir/.meltano/extractors/tap-postgres/venv/bin/tap-postgres', '--config', 'my_dir/.meltano/run/tap-postgres/tap.50ce392b-f1d4-4cdd-ac72-259851cd18b5.config.json']
2024-09-12T22:11:22.585409Z [debug    ] Created configuration at my_dir/.meltano/run/target-clickhouse/target.3f5a4c6e-2989-457f-b050-7eaabaf9f0d7.config.json
2024-09-12T22:11:22.591708Z [debug    ] Invoking: ['/my_dir/.meltano/loaders/target-clickhouse/venv/bin/target-clickhouse', '--config', 'my_dir/.meltano/run/target-clickhouse/target.3f5a4c6e-2989-457f-b050-7eaabaf9f0d7.config.json']
2024-09-12T22:11:22.612318Z [debug    ] waiting for process completion or exception
2024-09-12T22:11:24.000453Z [info     ] 2024-09-12 22:11:23,999 | INFO     | target-clickhouse    | Target 'target-clickhouse' is listening for input from tap. cmd_type=elb consumer=True job_name=dev:tap-postgres-to-target-clickhouse name=target-clickhouse producer=False run_id=f8218c74-2e16-4c66-9b07-375735a0b7d9 stdio=stderr string_id=target-clickhouse
2024-09-12T22:11:28.033748Z [info     ] {"type":"STATE","value":{}}    cmd_type=elb consumer=False job_name=dev:tap-postgres-to-target-clickhouse name=tap-postgres producer=True run_id=f8218c74-2e16-4c66-9b07-375735a0b7d9 stdio=stdout string_id=tap-postgres
2024-09-12T22:11:28.034544Z [info     ] 2024-09-12 22:11:28,033 | INFO     | tap-postgres         | Skipping deselected stream 'alerts-alert_ttl'. cmd_type=elb consumer=False job_name=dev:tap-postgres-to-target-clickhouse name=tap-postgres producer=True run_id=f8218c74-2e16-4c66-9b07-375735a0b7d9 stdio=stderr string_id=tap-postgres

...much skipped tables...

2024-09-12T22:11:28.243673Z [info     ] 2024-09-12 22:11:28,101 | INFO     | tap-postgres         | Skipping deselected stream 'my_db-my_table'. cmd_type=elb consumer=False job_name=dev:tap-postgres-to-target-clickhouse name=tap-postgres producer=True run_id=f8218c74-2e16-4c66-9b07-375735a0b7d9 stdio=stderr string_id=tap-postgres

...much skipped tables...

2024-09-12T22:38:06.649574Z [debug    ] head producer completed first as expected name=tap-postgres
2024-09-12T22:38:06.650730Z [debug    ] tail consumer is next block, wrapping up
2024-09-12T22:38:06.651529Z [info     ] 2024-09-12 22:38:06,651 | INFO     | target-clickhouse    | Target 'target-clickhouse' completed reading 1 lines of input (0 schemas, 0 records, 0 batch manifests, 1 state messages). cmd_type=elb consumer=True job_name=dev:tap-postgres-to-target-clickhouse name=target-clickhouse producer=False run_id=bc146224-34b7-4593-891e-b1130e41e69d stdio=stderr string_id=target-clickhouse
2024-09-12T22:38:06.654051Z [info     ] 2024-09-12 22:38:06,652 | INFO     | target-clickhouse    | Emitting completed target state {} cmd_type=elb consumer=True job_name=dev:tap-postgres-to-target-clickhouse name=target-clickhouse producer=False run_id=bc146224-34b7-4593-891e-b1130e41e69d stdio=stderr string_id=target-clickhouse
2024-09-12T22:38:06.704627Z [debug    ] Added to state dev:tap-postgres-to-target-clickhouse state payload {'singer_state': {}}
2024-09-12T22:38:06.734252Z [info     ] Incremental state has been updated at 2024-09-12 22:38:06.734191+00:00.
2024-09-12T22:38:06.735513Z [debug    ] Incremental state: {}         
2024-09-12T22:38:06.736648Z [info     ] {}                             cmd_type=elb consumer=True job_name=dev:tap-postgres-to-target-clickhouse name=target-clickhouse producer=False run_id=bc146224-34b7-4593-891e-b1130e41e69d stdio=stdout string_id=target-clickhouse
2024-09-12T22:38:06.890309Z [debug    ] Deleted configuration at my_dir/.meltano/run/tap-postgres/tap.18a2bc3b-53b5-459c-8615-1969b7058e83.config.json
2024-09-12T22:38:06.891923Z [debug    ] Deleted configuration at my_dir/.meltano/run/target-clickhouse/target.45121da9-2523-4fac-8479-0c49f5eb73f7.config.json
2024-09-12T22:38:06.911735Z [info     ] Block run completed.           block_type=ExtractLoadBlocks err=None set_number=0 success=True
Basically my_db-my_table skipped for some reason despite the meltano.yml configuration. Will be glad of any help
e
You should be using
select
instead.
tables
is not a setting supported by tap-postgres: https://docs.meltano.com/concepts/plugins/#select-extra
2
e
But I do not need full table select I need replication using WAL
e
I think you still need to set the
select
config, but maybe @visch knows better
👀 1
e
Also tried with added
Copy code
extractors:
  - name: tap-postgres
  select:
    - my_db-my_table.*
and result is same
v
@Egor Orekhov did you read https://github.com/MeltanoLabs/tap-postgres?tab=readme-ov-file#how-to-set-up-log-based-replication ? There's some steps for "turning on" log based for you table. Try those and let me know!
@Egor Orekhov read this too https://docs.meltano.com/concepts/plugins/#metadata-extra Let me know if you get stuck!