hey all, just completed my very first

meltano run

using

tap-s3-csv

and

target-postgres

. took me a bit to get my environment setup, and then had to battle some more issues with both plugins to get them properly installed, but was able to find the solutions searching through this slack and the docs. gotta say, super cool once it all came together. definitely see the power, and excited to learn more! one question I have at the moment is in regards to the

add_metadata_columns

config option for

target-postgres

. I set this to true, but am noticing some peculiar behavior with respect to

_sdc_batched_at

and

sdc_extracted_at

….while the original console logs show that the landing table is created with column data types of

timestamp without time zone

for both of these metadata columns, the time for

_sdc_batched_at

seems to land in the table in my current time (MT) and

_sdc_extracted_at

is landing in UTC. this 7 hr difference is rather confusing when looking at the table. any ideas what may be causing this? or thoughts on if this is likely a source system (s3) setting, local environment setting or meltano setting that I need to configure to get both metadata timestamps in the UTC time value?

I think it's a bug in target-postgres 😕

ah shoot, didn’t think to check the issues for the target, thanks. so your solution was just to set the

TZ

environment variable? did you set that within your

meltano.yml

at the

env

level for

target-postgres

so that it’s specific just to resolving this issue?

Actually I think I was only able to fix it running the target manually, which I think I forgot to put in the ticket 😞 If memory serves setting the env var and then running meltano caused other problems with meltano, but... it's been a while so I'm not really sure.

so it seems like I was actually able to solve this with the environment variable declared specific to just the problematic

target-postgres

. the difference from my earlier codeblock above is that I set TZ = UTC and not my local time.

plugins:
  loaders:
  - name: target-postgres
    variant: transferwise
    pip_url: pipelinewise-target-postgres
    env:
      TZ: UTC

this way when that line 68 is called in that init.py:

extended_record['_sdc_batched_at'] = datetime.now().isoformat()

it then plugs in the utc time (similar to a

datetime.utcnow().isoformat()

) & since

TZ

is only getting set to UTC within

target-postgres

, it doesn’t seem to be causing other problems with meltano (at least none that I have seen yet 🤞)