I ve seen a number of other threads talking about this issue Meltano #troubleshooting

I’ve seen a number of other threads talking about ...

josh_lloyd

07/14/2022, 6:11 PM

I’ve seen a number of other threads talking about this issue but none of them have provided a solution as far as I can tell. I’m trying to add a composite primary key to a stream that has no primary key at all by setting the

table-key-properties

metadata property as so:

Copy code

- name: tap-mysql--XXXX
    inherit_from: tap-mysql
    metadata:
      YYYY-conversionfileformats:
        table-key-properties:
        - conversionId
        - format

meltano verison 2.1.0 transferwise variant of tap-mysql I’ve tried every configuration of the names of the two desired primary key columns but every time I look at the

SCHEMA

message in the tap output the

key_properties

value is always

[]

. Thoughts?

edgar_ramirez_mondragon

07/14/2022, 6:31 PM

Hi @josh_lloyd, I’m seeing it’s an inherited plugin, so your case might be fixed by https://github.com/meltano/meltano/pull/6404.

edgar_ramirez_mondragon

07/14/2022, 6:32 PM

What is the output you see for

Copy code

meltano config --extras tap-mysql--XXXX

josh_lloyd

07/14/2022, 6:33 PM

Copy code

{
  "_catalog": null,
  "_state": null,
  "_load_schema": "tap_mysql__XXXX",
  "_select": [
    "YYYY-conversionfileformats.*"
  ],
  "_metadata": {
    "*": {
      "replication-method": "FULL_TABLE"
    }
  },
  "_schema": {},
  "_select_filter": [],
  "_metadata.*.replication-method": "FULL_TABLE",
  "_metadata.YYYY-conversionfileformats.table-key-properties": [
    "conversionId",
    "format"
  ]
}

josh_lloyd

07/14/2022, 6:33 PM

looks like they’re in there as expected

edgar_ramirez_mondragon

07/14/2022, 6:39 PM

It seems that

_metadata

does not include the keys you need, even though the flattened values are there. That does seem like the issue I linked above. Is it possible for you to install from the branch to test the fix or see if it’s a new and different issue? You can install the branch with

Copy code

pipx install --force --suffix="@6404" --python=python3.9 'meltano @ git+<https://github.com/meltano/meltano.git@6380-metadata-and-other-extras-of-kind-object-not-honored-when-using-inheritance>'

josh_lloyd

07/14/2022, 6:40 PM

sure, I’ll give it a shot

edgar_ramirez_mondragon

07/14/2022, 6:42 PM

cool, then you can run meltano as, for example

Copy code

meltano@6404 config --extras tap-mysql--XXXX

josh_lloyd

07/14/2022, 6:45 PM

cool they came through in the

SCHEMA

message and here is the output of the

config

command:

Copy code

{
  "_catalog": null,
  "_state": null,
  "_load_schema": "tap_mysql__XXXX",
  "_select": [
    "YYYY-conversionfileformats.*"
  ],
  "_metadata": {
    "*": {
      "replication-method": "FULL_TABLE"
    },
    "YYYY-conversionfileformats": {
      "table-key-properties": [
        "conversionId",
        "format"
      ]
    }
  },
  "_schema": {},
  "_select_filter": [],
  "_metadata.*.replication-method": "FULL_TABLE",
  "_metadata.YYYY-conversionfileformats.table-key-properties": [
    "conversionId",
    "format"
  ]
}

edgar_ramirez_mondragon

07/14/2022, 6:50 PM

Awesome!

josh_lloyd

07/14/2022, 6:51 PM

any idea on when this fix might get released?

edgar_ramirez_mondragon

07/14/2022, 7:02 PM

Next week for sure

Open in Slack

Previous Next