Hi, I'm using

tap-moosend

and I have a datetime conversion issue. If I run

meltano run tap-moosend target-jsonl

, I see datetime values like

"CreatedOn": "/Date(1625093153963+0000)/"

in the JSONL output. But if I change the target to something else, e.g.

target-csv

, I get the following error:

ValueError: Could not parse value '/Date(1625093153963+0000)/' for field 'CreatedOn'. cmd_type=elb consumer=True job_name=dev:tap-moosend-to-target-csv name=target-csv producer=False run_id=455f12d6-7426-424d-ac5f-04cee619bf45 stdio=stderr string_id=target-csv

Originally, I tried to develop a custom target and I got the same error. Since I just started learning Meltano, I tried the CSV target, so I think the error is not in my code. Do I need to create a transformer to parse this date (which is basically a unix timestamp)? Or is this a configuration issue on my side? Thanks!

Can you run

meltano invoke tap-moosend > out

and copy the schema line, and an example record with the data you have. It sounds like the schema is wrong for that record. The data needs to comply with the json schema given

I think the tap is at fault here, since it's probably communicating that

CreatedOn

is a date-time string while it doesn't actually conform to RFC 3339. One potential workaround is to override the schema for that field to make it just a string, without

format

:

extractors:
- name: tap-moosend
  schema:
    some_stream_id:
      CreatedOn:
        type: ["string", "null"]
        format: "none"  # a hack to make sure the current format is removed

https://docs.meltano.com/concepts/plugins/#schema-extra

schema for the

CreatedOn

field:

"CreatedOn": {"format": "date-time", "type": ["string", "null"]}

and the data in every record for this field:

"CreatedOn": "/Date(1625093153963+0000)/"

So I guess this is not compatible because the date is in Microsoft JSON date format. But in this case, why can I export it with

target-jsonl

but not with

target-csv

? And more importantly, what should I do?

tap-moosend

is from Meltano Hub, I use it to connect to the official Moosend API so I don't have control on the exported data and its format.

The tap has control of how the data is output so that's not fully true.

/Date(1625093153963+0000)/

is not a valid

date-time

, you can make it be a string like @Edgar Ramírez (Arch.dev) said. Or you can (I think appropriately) modify the date so it's a date-time in the tap to follow the schema properly

Yeah, I bet target-jsonl doesn't validate the

format

piece of the JSON schema so it's ok with it just being a valid string. I also don't see https://gitlab.com/hotglue/tap-moosend transforming that field, and the API docs confirm that they just send that: https://docs.moosend.com/developers/api-documentation/en/index-en.html. IMHO, the tap should be either 1) parsing those fields into valid date-time values or 2) Not declaring them to be date-time strings. As a workaround, my

schema

override suggestion above should let validation pass. Though, for example, the values will be stored as

"/Date(1465381164389)/"

in the CSV.

Thanks to both of you, all of these make sense and I'll turn off validation for this field. I might also fork the tap and send a PR (if I'll have the time) so the date-time fields will be properly formatted.