Curious what thoughts are on epoch date-time conversation this https://github.com/AutoIDM/tap-clickup/issues/54 Clickup API delivers date data as an epoch. We ended up going with delivering this data as a string to the target. (Maybe this should be an int, but I was probably a bit over concerned about https://en.wikipedia.org/wiki/Year_2038_problem , a quick test could have probably pushed me to using int) What do you think about converting this to

rfc3339

, we left this to the target as changing data felt wrong in the tap.

I would for sure use a standard format to save the info + timezone

We went with saving it as a string so we are keeping that data 🙏

I would convert it as the transformation to follow as much as possible the ELT pipeline

Got it so you would wait until it hits the DW and not do it in

tap-clickup

Correct

I think leaving as-is is a valid approach but a case can be made either way in the tap layer. I do think there's an ontological argument that the type is a UTC datetime, and type conformance at the tap layer is a reasonable action to perform before sending downstream. The other option is if we could introduce a JSON-schema

format

for epoch, which is clearly self-describing as (something like

datetime-epoch

) - which would then give the target guidance on how to load.

Thanks @aaronsteers great answer. Would the json-schema format be a meltano sdk thing or a spec thing?

I think it's both a tap developer decision and also a spec thing. JSON Schema spec allows a lot of freedom in

format

strings, aside from the ones already defined, I think anything is basically legal. But we'd need some kind of spec guidance and/or community alignment to get targets and taps on the same page with what the format string (or other JSON Schema annotation) is implying.

I get it now, I'm fairly green on the json spec stuff. Thank you!

I think some taps send already annotations like

format='sqltype:varchar(100)'

(not exactly but something like this) which tells the downstream target how the data was originally encoded at the tap level. But I don't know how common this is, and it probably wouldn't exactly match this use case because epoch isn't generally a SQL datatype - or if it is, it's

datetime

. 🙂