@taylor Hitting you up here as I didn't follow what you said in the bug report (https://github.com/meltano/meltano/issues/6282). When you say add to my plugin definition you mean to add that inside of meltano.yml under environments.dev.config.plugins.extractors.tap-postgres? And then to remove the previous lock for tap-postgres and then run

meltano lock tap-postgres

? That resulted in the same lock file which didn't have the capabilities I added to meltano.yml. So, there another plugin definition file I should be looking at?

@will_johnson I meant adding it to the global definition so you'd have:

plugins:
  extractors:
  - name: tap-postgres
    variant: transferwise
    pip_url: pipelinewise-tap-postgres
    capabilities:
      - properties
      - discover
      - state

but now that I think about it that may not be respected if you don't declare it as a custom plugin... @edgar_ramirez_mondragon am I mis-stating things here?

tap seems to support both, so that may not be an issue: https://github.com/transferwise/pipelinewise-tap-postgres/blob/14a0be55e8494f7f5b7ee4e6627ece902db6729a/tap_postgres/__init__.py#L380-L382 one thing I do notice in the yaml posted in the issue is the

metadata

is incorrectly nested inside

config

, while it should be at the same level (bump for #2964)

Ahhh, added it there and then relocked and that didn't seem to change anything. I can change the default_replication_method to LOG_BASED and set the replication-method to INCREMENTAL for the individual tables in the meantime.

Ah - I just remembered / found that we recently fixed this as well but need to better document. https://github.com/meltano/meltano/issues/6029 is the original issue

@edgar_ramirez_mondragon Thanks for pointing out my error with the level of the metadata!

I don't think that's it. The source code I think is looking for the dash https://github.com/transferwise/pipelinewise-tap-postgres/blob/14a0be55e8494f7f5b7ee4e6627ece902db6729a/tap_postgres/__init__.py#L72

Ahh, good find.

@will_johnson Looking at your logs in https://github.com/meltano/meltano/issues/6282#issuecomment-1165702100, I wonder if the

[null]

at the very end of the following SCHEMA message reveals the issue

{
  "type": "SCHEMA",
  "stream": "src_sample_store-widgets",
  "schema": {
    "type": "object",
    "properties": {
      "id": {
        "type": [
          "integer"
        ],
        "minimum": -2147483648,
        "maximum": 2147483647
      },
      "name": {
        "type": [
          "null",
          "string"
        ]
      },
      "widget_type": {
        "type": [
          "null",
          "string"
        ]
      },
      "created_at": {
        "type": [
          "null",
          "string"
        ],
        "format": "date-time"
      },
      "updated_at": {
        "type": [
          "null",
          "string"
        ],
        "format": "date-time"
      }
    },
    "definitions": {
      "sdc_recursive_integer_array": {
        "type": [
          "null",
          "integer",
          "array"
        ],
        "items": {
          "$ref": "#/definitions/sdc_recursive_integer_array"
        }
      },
      "sdc_recursive_number_array": {
        "type": [
          "null",
          "number",
          "array"
        ],
        "items": {
          "$ref": "#/definitions/sdc_recursive_number_array"
        }
      },
      "sdc_recursive_string_array": {
        "type": [
          "null",
          "string",
          "array"
        ],
        "items": {
          "$ref": "#/definitions/sdc_recursive_string_array"
        }
      },
      "sdc_recursive_boolean_array": {
        "type": [
          "null",
          "boolean",
          "array"
        ],
        "items": {
          "$ref": "#/definitions/sdc_recursive_boolean_array"
        }
      },
      "sdc_recursive_timestamp_array": {
        "type": [
          "null",
          "string",
          "array"
        ],
        "format": "date-time",
        "items": {
          "$ref": "#/definitions/sdc_recursive_timestamp_array"
        }
      },
      "sdc_recursive_object_array": {
        "type": [
          "null",
          "object",
          "array"
        ],
        "items": {
          "$ref": "#/definitions/sdc_recursive_object_array"
        }
      }
    }
  },
  "key_properties": [
    "id"
  ],
  "bookmark_properties": [
    null
  ]
}

Here you go

All right, so none of the metadata made it there. I was just looking at https://github.com/meltano/meltano/issues/3322 again, and I think that’s the same issue you’re seeing:

metadata

doesn’t work properly under

environments

.

I think that worked

You’re sure that’s a tap-postgres bug? The selection is making it to the generated catalog file properly?

I think so. I believe this shows what is happening: https://github.com/astrojuanlu/pipelinewise-tap-postgres/commit/d6d589899cf583b90c00f2efc553c8ded12caa66

Good catch, that’s definitely the same issue. @juan_luis_cano_rodriguez Did you manage to get in touch with the tap maintainers about whether they’d accept a narrow fix for that ignored

selected

metadata? A few of them of them are in this Slack as well. I think it’s definitely worth submitting a PR so there’s specific code to debate over.

I left a comment here discussing a bit the implications:

I understand the reasoning behind the current behavior and #58: if the source schema changes, there is potential data loss. However, now tap-postgres is "too smart" about it and we can't even choose how do we want to extract our data, which is annoying.

Would the maintainers accept a simpler PR that does not do a discovery phase if an explicit catalog is provided?

but so far no response. As you say, an alternative is to, rather than completely revert the "refresh schema always", do so in a way that at least the

selected

metadata is preserved. I'm happy to send a PR but I would not like to put the work without someone on the other side at least saying "yay, let's try"

@juan_luis_cano_rodriguez As long they describe pipelinewise-tap-postgres as a Singer tap and not something PipelineWise-specific, any behavior that goes against the Singer spec is a bug, so I’d expect them to be open to the contribution. If they think the ignore-selected behavior makes more sense for PipelineWise, they may request a boolean config option that lets users opt in our out of the standard Singer behavior. @peter_kosztolanyi Would you be open to the tap-postgres fix Juan described in his previous message in this thread?

heya, thanks for reaching me out. I'm currently not actively maintaining the repository and I can't give a good advise how to proceed. @samira_el what do you think is this something we can merge maybe, do we accept bugfix PRs?

As long they describe pipelinewise-tap-postgres as a Singer tap and not something PipelineWise-specific, any behavior that goes against the Singer spec is a bug

I personally agree with this sentence. Even if ppw taps are extended ones they are still meant to be compatible with the singer spec. Since #58 is merged, all changes in the catalog file are lost which were made manually or by external tools. I didn't look into the code but making this behaviour optional maybe should fix the problem? 🤷