Egor Orekhov
09/12/2024, 10:56 PMversion: 1
default_environment: dev
project_id: 6d246080-d283-4f3b-afe1-52264750c7c2
environments:
- name: dev
- name: staging
- name: prod
plugins:
extractors:
- name: tap-postgres
namespace: tap_postgres
variant: meltanolabs
pip_url: git+<https://github.com/MeltanoLabs/tap-postgres.git>
config:
port: 5432
user: my_user
password: my_password
database: my_db
replication_method: LOG_BASED
replication_slot: my_replication_slot
start_lsn: 0/0
tables:
- table: my_db.my_table
default_replication_method: LOG_BASED
host: my_host
loaders:
- name: target-clickhouse
variant: shaped-ai
namespace: target_clickhouse
pip_url: git+<https://github.com/shaped-ai/target-clickhouse.git>
config:
host: "my_host"
port: 8123
user: "my_user"
password: "my_password"
database: "my_db"
tables:
- table: "my_table"
Log I got:
meltano run tap-postgres target-clickhouse
2024-09-12T22:11:21.577887Z [debug ] meltano 3.5.1, Linux
2024-09-12T22:11:21.585315Z [debug ] /etc/timezone found, contents:
Etc/UTC
2024-09-12T22:11:21.586160Z [debug ] /etc/localtime found
2024-09-12T22:11:21.587621Z [debug ] 2 found:
{'/etc/timezone': 'Etc/UTC', '/etc/localtime is a symlink to': 'Etc/UTC'}
2024-09-12T22:11:21.597078Z [info ] Environment 'dev' is active
2024-09-12T22:11:21.625345Z [debug ] Creating DB engine for project at 'my_dir' with DB URI 'sqlite:my_dir'
2024-09-12T22:11:21.707162Z [debug ] found plugin in cli invocation plugin_name=tap-postgres
2024-09-12T22:11:21.708571Z [debug ] found plugin in cli invocation plugin_name=target-clickhouse
2024-09-12T22:11:21.709248Z [debug ] head of set is extractor as expected block=tap-postgres
2024-09-12T22:11:21.905256Z [debug ] found block block_type=loaders index=1
2024-09-12T22:11:21.905640Z [debug ] blocks idx=1 offset=0
2024-09-12T22:11:21.971992Z [debug ] found ExtractLoadBlocks set offset=0
2024-09-12T22:11:21.972442Z [debug ] All ExtractLoadBlocks validated, starting execution.
2024-09-12T22:11:22.130248Z [debug ] Skipped installing extractor 'tap-postgres'
2024-09-12T22:11:22.194198Z [debug ] Skipped installing loader 'target-clickhouse'
2024-09-12T22:11:22.194802Z [debug ] Installed 0/2 plugins
2024-09-12T22:11:22.195092Z [debug ] Skipped installing 2/2 plugins
2024-09-12T22:11:22.484866Z [debug ] Created configuration at my_dir/.meltano/run/tap-postgres/tap.50ce392b-f1d4-4cdd-ac72-259851cd18b5.config.json
2024-09-12T22:11:22.485981Z [debug ] Could not find tap.properties.json in my_dir/.meltano/extractors/tap-postgres/tap.properties.json, skipping.
2024-09-12T22:11:22.486638Z [debug ] Could not find tap.properties.cache_key in my_dir/.meltano/extractors/tap-postgres/tap.properties.cache_key, skipping.
2024-09-12T22:11:22.487188Z [debug ] Could not find state.json in my_dir/.meltano/extractors/tap-postgres/state.json, skipping.
2024-09-12T22:11:22.493536Z [debug ] Invoking: ['my_dir/.meltano/extractors/tap-postgres/venv/bin/tap-postgres', '--config', 'my_dir/.meltano/run/tap-postgres/tap.50ce392b-f1d4-4cdd-ac72-259851cd18b5.config.json']
2024-09-12T22:11:22.585409Z [debug ] Created configuration at my_dir/.meltano/run/target-clickhouse/target.3f5a4c6e-2989-457f-b050-7eaabaf9f0d7.config.json
2024-09-12T22:11:22.591708Z [debug ] Invoking: ['/my_dir/.meltano/loaders/target-clickhouse/venv/bin/target-clickhouse', '--config', 'my_dir/.meltano/run/target-clickhouse/target.3f5a4c6e-2989-457f-b050-7eaabaf9f0d7.config.json']
2024-09-12T22:11:22.612318Z [debug ] waiting for process completion or exception
2024-09-12T22:11:24.000453Z [info ] 2024-09-12 22:11:23,999 | INFO | target-clickhouse | Target 'target-clickhouse' is listening for input from tap. cmd_type=elb consumer=True job_name=dev:tap-postgres-to-target-clickhouse name=target-clickhouse producer=False run_id=f8218c74-2e16-4c66-9b07-375735a0b7d9 stdio=stderr string_id=target-clickhouse
2024-09-12T22:11:28.033748Z [info ] {"type":"STATE","value":{}} cmd_type=elb consumer=False job_name=dev:tap-postgres-to-target-clickhouse name=tap-postgres producer=True run_id=f8218c74-2e16-4c66-9b07-375735a0b7d9 stdio=stdout string_id=tap-postgres
2024-09-12T22:11:28.034544Z [info ] 2024-09-12 22:11:28,033 | INFO | tap-postgres | Skipping deselected stream 'alerts-alert_ttl'. cmd_type=elb consumer=False job_name=dev:tap-postgres-to-target-clickhouse name=tap-postgres producer=True run_id=f8218c74-2e16-4c66-9b07-375735a0b7d9 stdio=stderr string_id=tap-postgres
...much skipped tables...
2024-09-12T22:11:28.243673Z [info ] 2024-09-12 22:11:28,101 | INFO | tap-postgres | Skipping deselected stream 'my_db-my_table'. cmd_type=elb consumer=False job_name=dev:tap-postgres-to-target-clickhouse name=tap-postgres producer=True run_id=f8218c74-2e16-4c66-9b07-375735a0b7d9 stdio=stderr string_id=tap-postgres
...much skipped tables...
2024-09-12T22:38:06.649574Z [debug ] head producer completed first as expected name=tap-postgres
2024-09-12T22:38:06.650730Z [debug ] tail consumer is next block, wrapping up
2024-09-12T22:38:06.651529Z [info ] 2024-09-12 22:38:06,651 | INFO | target-clickhouse | Target 'target-clickhouse' completed reading 1 lines of input (0 schemas, 0 records, 0 batch manifests, 1 state messages). cmd_type=elb consumer=True job_name=dev:tap-postgres-to-target-clickhouse name=target-clickhouse producer=False run_id=bc146224-34b7-4593-891e-b1130e41e69d stdio=stderr string_id=target-clickhouse
2024-09-12T22:38:06.654051Z [info ] 2024-09-12 22:38:06,652 | INFO | target-clickhouse | Emitting completed target state {} cmd_type=elb consumer=True job_name=dev:tap-postgres-to-target-clickhouse name=target-clickhouse producer=False run_id=bc146224-34b7-4593-891e-b1130e41e69d stdio=stderr string_id=target-clickhouse
2024-09-12T22:38:06.704627Z [debug ] Added to state dev:tap-postgres-to-target-clickhouse state payload {'singer_state': {}}
2024-09-12T22:38:06.734252Z [info ] Incremental state has been updated at 2024-09-12 22:38:06.734191+00:00.
2024-09-12T22:38:06.735513Z [debug ] Incremental state: {}
2024-09-12T22:38:06.736648Z [info ] {} cmd_type=elb consumer=True job_name=dev:tap-postgres-to-target-clickhouse name=target-clickhouse producer=False run_id=bc146224-34b7-4593-891e-b1130e41e69d stdio=stdout string_id=target-clickhouse
2024-09-12T22:38:06.890309Z [debug ] Deleted configuration at my_dir/.meltano/run/tap-postgres/tap.18a2bc3b-53b5-459c-8615-1969b7058e83.config.json
2024-09-12T22:38:06.891923Z [debug ] Deleted configuration at my_dir/.meltano/run/target-clickhouse/target.45121da9-2523-4fac-8479-0c49f5eb73f7.config.json
2024-09-12T22:38:06.911735Z [info ] Block run completed. block_type=ExtractLoadBlocks err=None set_number=0 success=True
Basically my_db-my_table skipped for some reason despite the meltano.yml configuration.
Will be glad of any helpEdgar Ramírez (Arch.dev)
09/12/2024, 11:06 PMselect
instead. tables
is not a setting supported by tap-postgres:
https://docs.meltano.com/concepts/plugins/#select-extraEgor Orekhov
09/12/2024, 11:08 PMEdgar Ramírez (Arch.dev)
09/12/2024, 11:09 PMselect
config, but maybe @visch knows betterEgor Orekhov
09/13/2024, 2:32 PMextractors:
- name: tap-postgres
select:
- my_db-my_table.*
and result is samevisch
09/13/2024, 4:14 PMvisch
09/13/2024, 4:27 PM