https://linen.dev logo
Join Slack
Powered by
Hello, I am testing out Meltano and trying to crea...
# troubleshooting
r
Hello, I am testing out Meltano and trying to create a connector using tap-postgres and target-snowflake (both meltanolabs variant) and using an S3 for the backend state. Connection to source and target is working well, the data is fetched, but it is always doing 
full_table
syncs while declared as 
log_based
. It seems that the S3 backend state is always initiated to the initial, empty, state.
Copy code
completed	
  singer_state	
    bookmarks	
      public-table_1	{}
      public-table_2	{}
      public-table_3	{}
      public-table_4	{}
partial	{}
At the end of the data ingestion for a single table, 
{"starting_replication_value": null}
gets added to the table, but then again overwritten as 
STATE
message by the connector. The source DB has a replication slot, that is (obviously) not consumed and has a publication, which does not get referenced by tap-postgres as per docs there is no way to set it. Is there something I am missing with the setup here?
The initial issue is with 
wal2json
replication slot, with a 
pgoutput
slot the error I get is this:
Copy code
psycopg2.errors.InternalError_: unrecognized pgoutput option: format-version 
CONTEXT:  slot "replication_slot_name", output plugin "pgoutput", in the startup callback
Postgres DB is running on 15.12
I do see a hard-coded option for the `logical_replication_cursor.start_replication()`:
Copy code
options={
                "format-version": 2,
                "include-transaction": False,
                "add-tables": self.fully_qualified_name,
            },
Is there a way to change it via config in YML?
e
You mean 
format-version
?
Also, what does your 
meltano.yml
look like?
r
Yes, the main issue seems to be the 
format-version
option, I will try to see if the others "fit" as well
meltano.yml
looks like this:
Copy code
version: 1
default_environment: dev
project_id: {some_id}
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-postgres
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/tap-postgres>
    metadata:
      "*":
        replication_method: LOG_BASED
        replication_key: _sdc_lsn
    select:
      - public-table1.column1
      - public-table1.column2
      - public-table1.column3
      - public-table1.column4
      - public-table1.column5
      - public-table1.column6
      - public-taböe1.column7
      - public-table1.column8
      - public-table1.column9

  loaders:
  - name: target-snowflake
    variant: meltanolabs
    pip_url: meltanolabs-target-snowflake
  utilities:
  - name: cron
    variant: meltano
    pip_url: git+<https://github.com/meltano/cron-ext.git>
jobs:
- name: test_job
  tasks:
  - tap-postgres target-snowflake
schedules:
- name: test_job_schedule
  interval: '*/15 * * * *'
  job: test_job
3 Views
Open in Slack
PreviousNext
Powered by