nidhi_kakulawaram
10/03/2023, 9:07 AMmeltano run tap-snowflake target-s3
. For imports without dates, everything works great. However, for some reason when there is a DATE type in Snowflake like 1998-01-01, tap-snowflake converts it to 1998-01-01T000000+00:00. I tested this by invoking just the tap-snowflake and storing the results in a json. This causes the upload of the data to target-s3 to error due to data type mismatch. It is expecting a date type but gets a datetime object. I am not sure of the best way to handle this date/datetime type mismatch. Could I get some suggestions/assisstance? I was thinking either fixing the tap-snowflake (I have been unable to isolate where this transformation is occurring) or making target-s3 more robust to handle date vs datetime. For further context, here is my error message:
2023-10-03T07:45:39.979528Z [info ] jsonschema.exceptions.ValidationError: '1997-08-16T00:00:00+00:00' is not a 'date' cmd_type=elb consumer=True name=table-specifier-cede371e-e0db-41b5-8340-98505fe6475d-loader producer=False stdio=stderr string_id=table-specifier-cede371e-e0db-41b5-8340-98505fe6475d-loader
2023-10-03T07:45:39.979590Z [info ] cmd_type=elb consumer=True name=table-specifier-cede371e-e0db-41b5-8340-98505fe6475d-loader producer=False stdio=stderr string_id=table-specifier-cede371e-e0db-41b5-8340-98505fe6475d-loader
2023-10-03T07:45:39.979642Z [info ] Failed validating 'format' in schema['properties']['web_rec_start_date']: cmd_type=elb consumer=True name=table-specifier-cede371e-e0db-41b5-8340-98505fe6475d-loader producer=False stdio=stderr string_id=table-specifier-cede371e-e0db-41b5-8340-98505fe6475d-loader
2023-10-03T07:45:39.979693Z [info ] {'format': 'date', 'type': ['string', 'null']} cmd_type=elb consumer=True name=table-specifier-cede371e-e0db-41b5-8340-98505fe6475d-loader producer=False stdio=stderr string_id=table-specifier-cede371e-e0db-41b5-8340-98505fe6475d-loader
2023-10-03T07:45:39.979743Z [info ] cmd_type=elb consumer=True name=table-specifier-cede371e-e0db-41b5-8340-98505fe6475d-loader producer=False stdio=stderr string_id=table-specifier-cede371e-e0db-41b5-8340-98505fe6475d-loader
2023-10-03T07:45:39.979797Z [info ] On instance['web_rec_start_date']: cmd_type=elb consumer=True name=table-specifier-cede371e-e0db-41b5-8340-98505fe6475d-loader producer=False stdio=stderr string_id=table-specifier-cede371e-e0db-41b5-8340-98505fe6475d-loader
2023-10-03T07:45:39.979846Z [info ] '1997-08-16T00:00:00+00:00' cmd_type=elb consumer=True name=table-specifier-cede371e-e0db-41b5-8340-98505fe6475d-loader producer=False stdio=stderr string_id=table-specifier-cede371e-e0db-41b5-8340-98505fe6475d-loader
2023-10-03T07:45:41.736450Z [error ] Loader failed
2023-10-03T07:45:41.736638Z [error ] Block run completed.
user
10/03/2023, 1:46 PMnidhi_kakulawaram
10/03/2023, 2:48 PMnidhi_kakulawaram
10/03/2023, 7:40 PM