Hello, I am facing an issue with transforming a date value in my Meltano pipeline. I'm using

tap-bigquery

(anelendata) and

target-clickhouse

(shaped-ai) to extract data from BigQuery and load it into ClickHouse. • I'm selecting two columns from BigQuery:

mydate

(Date) and

myint

(Integer). • I'm applying a

stream_maps

transformation on the target side as follows:

stream_maps:

mystream:

final_mydate: mydate

final_mynewdate: ~"str(datetime.datetime.strptime(mydate, '%Y-%m-%dT%H:%M:%S') + datetime.timedelta(days=myint))"~

(datetime.datetime.strptime(mydate, '%Y-%m-%dT%H:%M:%S') + datetime.timedelta(days=myint)).date()

The final_mydate stays as Date type on Clickhouse table but the final_mynewdate changes to String. I’ve tested both scenarios: 1. Allowing Meltano to create the table automatically. 2. Using an existing table with the desired data types. In both cases,

final_mynewdate

is stored as a

String

. Any thoughts on how could I resolve that? Is it related to that bug ? thank you in advance :)

Can you help me understand?

final_mynewdate

is coming out as a string, but it looks like your mapper asks for it to be a string?

final_mynewdate: "str(....)"

Am I missing something?

You are absolutely right, my bad, I accidentally pasted a previous version, This is my final transformation

(datetime.datetime.strptime(mydate, '%Y-%m-%dT%H:%M:%S') + datetime.timedelta(days=myint)).date()

I'd let @Edgar Ramírez (Arch.dev) confirm but it does look like a textbook example of your linked issue. The issue title should probably say 'are typed as strings'.

Yes, that seems to be the linked bug! (Updated the title too 😅)