Hi all having some trouble using stream maps to filter schem Meltano #plugins-general

Hi all, having some trouble using stream_maps to f...

ryan_whitten

05/03/2022, 8:43 PM

Hi all, having some trouble using stream_maps to filter schema messages. According to the docs, setting a field to null should remove it from both the RECORD and SCHEMA messages. I'm only seeing it remove the field from RECORD messages. Is this expected or a bug? Example config:

Copy code

"stream_maps": {
  "bounces": {
    "INSERTTS": null
  }
}

Resulting schema:

Copy code

{
  "type": "SCHEMA",
  "stream": "bounces",
  "schema": {
    "properties": {
      "ACCTCODE": {
        "type": [
          "number",
          "null"
        ]
      },
      "OYBACCTCODE": {
        "type": [
          "number",
          "null"
        ]
      },
      "JOBCODE": {
        "type": [
          "number",
          "null"
        ]
      },
      "BATCHCODE": {
        "type": [
          "number",
          "null"
        ]
      },
      "SUBSCRIBERCODE": {
        "type": [
          "string",
          "null"
        ]
      },
      "SUBSCRIBERKEY": {
        "type": [
          "string",
          "null"
        ]
      },
      "EVENTDATE": {
        "type": [
          "string",
          "null"
        ]
      },
      "BOUNCECATCODE": {
        "type": [
          "number",
          "null"
        ]
      },
      "BOUNCECAT": {
        "type": [
          "string",
          "null"
        ]
      },
      "BOUNCESUBCATCODE": {
        "type": [
          "number",
          "null"
        ]
      },
      "BOUNCESUBCAT": {
        "type": [
          "string",
          "null"
        ]
      },
      "BOUNCETYPECODE": {
        "type": [
          "number",
          "null"
        ]
      },
      "BOUNCETYPE": {
        "type": [
          "string",
          "null"
        ]
      },
      "SMTPMSG": {
        "type": [
          "string",
          "null"
        ]
      },
      "SMTPCODE": {
        "type": [
          "number",
          "null"
        ]
      },
      "SMTPBOUNCEREASON": {
        "type": [
          "string",
          "null"
        ]
      },
      "SENT_UUID": {
        "type": [
          "string",
          "null"
        ]
      },
      "APPLICATIONID": {
        "type": [
          "string",
          "null"
        ]
      },
      "ACCOUNTNUMBER": {
        "type": [
          "string",
          "null"
        ]
      },
      "LEADID": {
        "type": [
          "string",
          "null"
        ]
      },
      "INSERTTS": {
        "format": "date-time",
        "type": [
          "string",
          "null"
        ]
      }
    },
    "type": "object"
  },
  "key_properties": []
}

Resulting record:

Copy code

{
  "type": "RECORD",
  "stream": "bounces",
  "record": {
    "ACCTCODE": 7217054,
    "OYBACCTCODE": null,
    "JOBCODE": 17767768,
    "BATCHCODE": 442,
    "SUBSCRIBERCODE": "xxx",
    "SUBSCRIBERKEY": "<mailto:xxx@xxx.com|xxx@xxx.com>",
    "EVENTDATE": "2022-05-02 06:16:36",
    "BOUNCECATCODE": 1,
    "BOUNCECAT": "xxx",
    "BOUNCESUBCATCODE": 2001,
    "BOUNCESUBCAT": "xxx",
    "BOUNCETYPECODE": 0,
    "BOUNCETYPE": "xxx",
    "SMTPMSG": "xxx",
    "SMTPCODE": 552,
    "SMTPBOUNCEREASON": "xxx",
    "SENT_UUID": "",
    "APPLICATIONID": "",
    "ACCOUNTNUMBER": null,
    "LEADID": ""
  },
  "time_extracted": "2022-05-03T20:25:32.844146Z"
}

ryan_whitten

05/03/2022, 9:02 PM

After some digging and finding this commit, I think that only fixed half the problem of the original issue but leaves the field in

transformed_schema["properties"]

. If I add

transformed_schema["properties"].pop(prop_key, None)

in place of the changed line to my local package, it works as intended. I could be missing something about the intended behavior, but seems like it's a bug? cc @edgar_ramirez_mondragon

edgar_ramirez_mondragon

05/03/2022, 9:11 PM

Hi @ryan_whitten. Thanks for digging into the code. I think indeed the expected behavior is for SCHEMA messages to be dropped too. Can you file an issue in the sdk repo? Maybe you're also willing to send us a merge request 😄

ryan_whitten

05/03/2022, 9:14 PM

Cool, thanks! Yeah I will open an issue and can probably create a MR in the next couple days to fix

edgar_ramirez_mondragon

05/03/2022, 9:14 PM

Thank you!

ryan_whitten

05/03/2022, 9:21 PM

Opened https://gitlab.com/meltano/sdk/-/issues/370

Open in Slack

Previous Next