If I wanted to add `TIMESTAMP` type support to `ta...
# singer-targets
e
If I wanted to add
TIMESTAMP
type support to
target-parquet
, how do I signal via the Singer Spec properly? I've successfully upgraded pyarrow to
10.0.1
and the output to
2.6
parquet spec.. but it's not quite bringing to bear the needed output References : https://github.com/estrategiahq/target-parquet/issues/12
DateTimeType
is outputting string
TimeType
might be a path forward but.. it's input is a string which I do not quite understand
a
e
thank you @aaronsteers! indeed this helps and I will give it a shot now in my fork .. I've got a sample file of the new spec as well so.. something I can shoot for so to speak to get support in TRINO v405+ .. lot of optimizations apparently there so will try my best to get this upstreamed
a
Nice 👍
e
I think the issue now is in the Singer Target for parquet.. I realized that I should be able to call out DateTime types from a tap.. and it should (now that they've merged my PR to upgrade
pyarrow == 10.0.1
) .. but that logic will need time to test/perfect
I can understand why it's behaving like this, since pre2.x parquet didn't have these types.. it just treated datetime stamps as a LONG and nothing more