https://linen.dev logo
Join Slack
Powered by
I am starting to track down some datetime discrepa...
# troubleshooting
e
I am starting to track down some datetime discrepancies in my data.. and the culprit might be something inside airflow/meltano... in Ubuntu I set the timezone to CEST .. and the time is reflected correctly.. but when I trigger a pipeline I am seeing this
Copy code
2022-10-02T15:14:20.281830Z [info     ] time=2022-10-02 17:14:20 name=...
Notice this leftmost datetime? Where is this set? It seems to be off by 2 hours
guessing this is some.. python or flask ? setting
Docs seem to come up empty
c
The two time strings are different. The first one is a time stamp that includes a time zone, the second one is a time stamp without a timezone indicator. To me this line looks correct.
e
thanks @christoph, But is that timezone.. UTC in the first left-most ? the real local time was 17:14 .. from where is this 15:14 ?
I am trying to make a note, when data has been sourced .. via a single column for each row.. and the time coming out of HIVE is off by quite a bit.. making me try to track down all the time settings i can
c
The first timestamp has the Zulu time timezone (
Z
) suffix. Zulu time is the timezone that has no offset from UTC.
e
so that is.. in effect.. UTC time?
I'll need to debug then, if that time is supposed to be UTC.. when this python code in the tap is yielding a different time
Copy code
def generate_timestamp(requested_timestamp_type):

    if requested_timestamp_type == 'hive':
        # TRINO and HIVE requires this format for native timestamps in parquet
        return round(int(datetime.utcnow().timestamp()*1000000), -3)
    else:
        # POSTGRESQL
        return datetime.utcnow().isoformat() + 'Z'
Thanks for the checkin
c
Yes. I think scientifically, it's actually not correct to refer to the 
Z
as "Zulu time", but yes, it's 
UTC
(not to be confused with 
GMT
)
e
once it hits hive.. the time is even MORE changed.. so.. I guess somewhere something is off
c
I use 
pendulum
as a helper library (Meltano SDK already depends on 
pendulum
) for datetime functionality.
e
nice to know! thank you
2 Views
Open in Slack
PreviousNext
Powered by