I m using `tap postgres` `meltanolabs` variant and `target b Meltano #singer-targets

I'm using `tap-postgres` (`meltanolabs` variant) a...

TomasB

01/29/2025, 3:47 PM

I'm using

tap-postgres

(

meltanolabs

variant) and

target-bigquery

(

youcruit

variant) . Having an issue when the column datatype from postgres side is

date

. Getting the below error

Copy code

joblib.externals.loky.process_executor._RemoteTraceback:                     
Traceback (most recent call last): 
File "/project/.meltano/loaders/target-bigquery/venv/lib/python3.10/site-packages/joblib/_utils.py", line 72, in __call__ 
    return self.func(**kwargs) 
   File "/project/.meltano/loaders/target-bigquery/venv/lib/python3.10/site-packages/joblib/parallel.py", line 598, in __call__ 
     return [func(*args, **kwargs) 
   File "/project/.meltano/loaders/target-bigquery/venv/lib/python3.10/site-packages/joblib/parallel.py", line 598, in <listcomp> 
     return [func(*args, **kwargs) 
   File "/project/.meltano/loaders/target-bigquery/venv/lib/python3.10/site-packages/singer_sdk/target_base.py", line 470, in _drain_sink 
     self.drain_one(sink)       
   File "/project/.meltano/loaders/target-bigquery/venv/lib/python3.10/site-packages/singer_sdk/target_base.py", line 460, in drain_one 
     sink.process_batch(draining_status) 
   File "/project/.meltano/loaders/target-bigquery/venv/lib/python3.10/site-packages/target_bigquery/sinks.py", line 354, in process_batch 
     writer(tempfile, self.parsed_schema, avro_records) 
   File "fastavro/_write.pyx", line 796, in fastavro._write.writer 
   File "fastavro/_write.pyx", line 738, in fastavro._write.Writer.write 
   File "fastavro/_write.pyx", line 465, in fastavro._write.write_data 
   File "fastavro/_write.pyx", line 409, in fastavro._write.write_record 
   File "fastavro/_write.pyx", line 463, in fastavro._write.write_data 
   File "fastavro/_write.pyx", line 343, in fastavro._write.write_union 
 ValueError: datetime.datetime(2024, 12, 17, 0, 0) (type <class 'datetime.datetime'>) do not match ['null', 'string'] on field <column_name>

and fails the sync. Is it possible to overwrite the datatype on bigquery side?

✅ 1

Edgar Ramírez (Arch.dev)

01/29/2025, 10:50 PM

You can probably set the

schema

for the tap to override that field to be a

type: string

. https://docs.meltano.com/concepts/plugins/#schema-extra

TomasB

01/29/2025, 10:52 PM

ahh I see. Will try that. Thank you!

TomasB

01/31/2025, 9:51 PM

Thank you for the suggestion, that worked. I ran into another issue though. Some of the records are "NULL" and for the column that has "NULL" records I get the error

Copy code

jsonschema.exceptions.ValidationError: None is not of type 'string'
Failed validating 'type' in schema['properties']['column_name']
On instance['column_name']:
None

Would there be any work around besides not having "NULL" values in the column?

Edgar Ramírez (Arch.dev)

01/31/2025, 9:56 PM

Ah, yes

Copy code

type: [string, "null"]

(notice the quotes)

TomasB

02/03/2025, 9:15 PM

That fixed it. Thank you for your quick response!

🙌 1

3 Views

Open in Slack

Previous Next