I am using the meltanolabs variant of

tap-postgres

. When a

meltano run tap-postgres target-xyz

breaks mid-run, I’m not seeing any partial state records stored, when I check

meltano state get {STATE_ID}

. In reviewing the docs [here](https://docs.meltano.com/guide/integration#internal-state-merge-logic), I believe that the meltanolabs variant should emit state, but am confused about why it doesn’t show up.

Could you show your meltano.yml? It should be working

Thanks Derek. Here’s the yaml:

version: 1
default_environment: dev
project_id: 56cb011e-0de6-4256-ac45-5f0b5c938d54
environments:
- name: dev
plugins:
  extractors:
  - name: tap-postgres
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/tap-postgres.git>
    select:
    - '!*.email'
    - '!*.email_display'
    - '!*.firstname'
    - '!*.lastname'
    - '!*.password_hash'
    - '!*.signing_key'
    - public-users.*
    - public-account_invites.*
    - public-accounts.*
    - public-user_roles.*
    - public-product_licenses.*
    - public-teams.*
    - public-user_teams.*
  - name: tap-postgres--documents
    inherit_from: tap-postgres
    select:
    - public-documents.*

  - name: tap-postgres--document-edit
    inherit_from: tap-postgres
    select:
    - public-document_edit.*

  - name: tap-postgres--sessions
    inherit_from: tap-postgres
    select:
    - public-sessions.*
  loaders:
  - name: target-athena
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/target-athena.git>
    config:
      athena_database: meltano_postgres
      aws_region: us-east-1
      s3_bucket: bucket-name
      s3_staging_dir: <s3://staging_dir/temp/>
      add_record_metadata: true
      encryption_type: none
      compression: gzip
      object_format: jsonl

I think the issue is that you're not setting the stream to incremental and specifying the replication key. So maybe let's pick one table as an example.

public-team

To set the replication-key you have to do something like this

extractors:
  - name: tap-postgres
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/tap-postgres.git>
    select:
    - '!*.email'
    - '!*.email_display'
    - '!*.firstname'
    - '!*.lastname'
    - '!*.password_hash'
    - '!*.signing_key'
    - public-users.*
    - public-account_invites.*
    - public-accounts.*
    - public-user_roles.*
    - public-product_licenses.*
    - public-teams.*
    - public-user_teams.*
    metadata:
      'public-team':
        replication-method: INCREMENTAL
        replication_key: "updated_at"

Right now if you ran

meltano invoke --dump=catalog tap-postgres > catalog.json

I think you'd see your tables are getting pulled with the relication method of FULL_TABLE

Aha, that makes it clearer @visch. Thank you. I’ll work on shifting the relevant entities to INCREMENTAL. Just to be crystal clear, a FULL_TABLE replication will always copy the entirety of the table even if there was a previous run that broke mid-run?

yes!