Hello Tap Devs My tap is currently saving a timestamp in str Meltano #singer-tap-development

Hello Tap Devs! My tap is currently saving a tim...

emcp

09/30/2022, 5:13 PM

Hello Tap Devs! My tap is currently saving a timestamp in string format

YYYY-MM-DDThh-mm-SS.ssssssZ

This is because that is what my usual target postgresql wants... but NOW I am playing with trino and its integration with HIVE Metastore to use S3 as a datalake that I can query... and there the timestamp gets rejected unless I give it in

MSfromepoch

format with a precision of 3 places removed ie

1663529000000000

emcp

09/30/2022, 5:13 PM

should my TAP be somehow helping here.. or is it up to the Target to decipher these fields ? I want to support both postgres and parquet targets but the timestamps would in theory need to be different if I want them to seamlessly get handled via tap+target .. right?

aaronsteers

09/30/2022, 5:39 PM

We've had related discussions in a few places but the general guidance is to pass the data using standard json/python types for datetime (depending upon what layer we're referring to), and let the tap or target translate as needed according to each's preferred conventions. Does this help?

emcp

10/01/2022, 11:00 AM

I think I can of course.. fork the parquet target or postgres.. but this is an interesting dilemma .. I guess.. I guess I could put a flag / option in my TAP .. on what format the query timestamp should take..

emcp

10/01/2022, 11:00 AM

and documentation points out that fact and users are off to the races from there?

emcp

10/02/2022, 1:14 PM

unfortunately I discovered that I need to actually change the Singer Spec (the model of the Tap) itself to support HIVE natively

emcp

10/02/2022, 1:52 PM

from

th.DatetimeType

th.IntegerType

emcp

10/02/2022, 1:53 PM

I guess if I fork the

target-parquet

and change the output behavior it might work .. but unclear if it could be merged upstream or if that's desirable

Open in Slack

Previous Next