I am running into an issue when trying to schema extra to change the type of a column in a stream for tap-bls. In the tap’s catalog the type is number and I am trying to force it to be a string. When I dump the catalog it appears that my changes are in it, however, I get a type error when I run the tap.

CRITICAL could not convert string to float: '-' cmd_type=extractor job_id=2022-02-14T153203--tap-bls--target-s3-jsonl-bls name=tap-bls run_id=82638c7b-a0c0-4220-945b-28937ef0df4c stdio=stderr
2022-02-14T15:33:20.558368Z [info     ] Traceback (most recent call last): cmd_type=extractor job_id=2022-02-14T153203--tap-bls--target-s3-jsonl-bls name=tap-bls run_id=82638c7b-a0c0-4220-945b-28937ef0df4c stdio=stderr

Hi @chris_marchetti! The catalog might be taking your override, but the tap's schema messages may not. Can you check the output of

meltano invoke tap-bls

? You'll see the raw singer messages like

{"type": "SCHEMA", "stream": "my_stream", "schema": {...}}
{"type": "RECORD", "stream": "my_stream", "record": {...}}

The

schema

value of the

SCHEMA

message should reveal if the tap is actually using the override or ignoring it. Taps built with the SDK do come with this feature out-of-the-box, but https://github.com/frasermarlow/tap-bls might not

@edgar_ramirez_mondragon Thanks for your reply. I ran

meltano invoke tap-bls

and it seems to be behaving the way I expect, but it still is erroring in the same way.

{
  "type": "SCHEMA",
  "stream": "LASST720000000000004",
  "schema": {
    "properties": {
      "SeriesID": {
        "type": [
          "null",
          "string"
        ]
      },
      "year": {
        "type": [
          "null",
          "string"
        ]
      },
      "period": {
        "type": [
          "null",
          "string"
        ]
      },
      "value": {
        "type": [
          "null",
          "string"
        ]
      },
      "footnotes": {
        "type": [
          "null",
          "string"
        ]
      },
      "full_period": {
        "format": "date-time",
        "type": [
          "null",
          "string"
        ]
      },
      "time_extracted": {
        "format": "date-time",
        "type": [
          "null",
          "string"
        ]
      },
      "month": {
        "type": [
          "null",
          "integer"
        ]
      },
      "net_change_1": {
        "type": [
          "null",
          "number"
        ]
      },
      "net_change_3": {
        "type": [
          "null",
          "number"
        ]
      },
      "net_change_6": {
        "type": [
          "null",
          "number"
        ]
      },
      "net_change_12": {
        "type": [
          "null",
          "number"
        ]
      },
      "pct_change_1": {
        "type": [
          "null",
          "number"
        ]
      },
      "pct_change_3": {
        "type": [
          "null",
          "number"
        ]
      },
      "pct_change_6": {
        "type": [
          "null",
          "number"
        ]
      },
      "pct_change_12": {
        "type": [
          "null",
          "number"
        ]
      }
    },
    "type": [
      "null",
      "object"
    ],
    "additionalProperties": [
      "schema",
      "record",
      "type",
      "stream"
    ]
  },
  "key_properties": [
    "year"
  ]
}

The value property is the one that is failing.

"value": {"type": ["null", "string"]},

Interesting, so it seems like the target is forcing the conversion? @chris_marchetti What's the variant of target-s3-jsonl you're using?

The pip url is

target-s3-jsonl==1.0.0

Ok, so diving in the tap code, it looks like the conversion is hardcoded: https://github.com/frasermarlow/tap-bls/blob/58c6204558da464d7b266be1e91a64f2a9ea0e41/tap_bls/sync.py#L152. I'd file an issue in the tap repo and/or contact Fraser (he's in the Singer slack)

Thanks! I appreciate the help.