Hi, I am running `meltano run tap-snowflake target...
# troubleshooting
n
Hi, I am running
meltano run tap-snowflake target-s3
. For imports without dates, everything works great. However, for some reason when there is a DATE type in Snowflake like 1998-01-01, tap-snowflake converts it to 1998-01-01T000000+00:00. I tested this by invoking just the tap-snowflake and storing the results in a json. This causes the upload of the data to target-s3 to error due to data type mismatch. It is expecting a date type but gets a datetime object. I am not sure of the best way to handle this date/datetime type mismatch. Could I get some suggestions/assisstance? I was thinking either fixing the tap-snowflake (I have been unable to isolate where this transformation is occurring) or making target-s3 more robust to handle date vs datetime. For further context, here is my error message:
Copy code
2023-10-03T07:45:39.979528Z [info     ] jsonschema.exceptions.ValidationError: '1997-08-16T00:00:00+00:00' is not a 'date' cmd_type=elb consumer=True name=table-specifier-cede371e-e0db-41b5-8340-98505fe6475d-loader producer=False stdio=stderr string_id=table-specifier-cede371e-e0db-41b5-8340-98505fe6475d-loader
2023-10-03T07:45:39.979590Z [info     ]                                cmd_type=elb consumer=True name=table-specifier-cede371e-e0db-41b5-8340-98505fe6475d-loader producer=False stdio=stderr string_id=table-specifier-cede371e-e0db-41b5-8340-98505fe6475d-loader
2023-10-03T07:45:39.979642Z [info     ] Failed validating 'format' in schema['properties']['web_rec_start_date']: cmd_type=elb consumer=True name=table-specifier-cede371e-e0db-41b5-8340-98505fe6475d-loader producer=False stdio=stderr string_id=table-specifier-cede371e-e0db-41b5-8340-98505fe6475d-loader
2023-10-03T07:45:39.979693Z [info     ]     {'format': 'date', 'type': ['string', 'null']} cmd_type=elb consumer=True name=table-specifier-cede371e-e0db-41b5-8340-98505fe6475d-loader producer=False stdio=stderr string_id=table-specifier-cede371e-e0db-41b5-8340-98505fe6475d-loader
2023-10-03T07:45:39.979743Z [info     ]                                cmd_type=elb consumer=True name=table-specifier-cede371e-e0db-41b5-8340-98505fe6475d-loader producer=False stdio=stderr string_id=table-specifier-cede371e-e0db-41b5-8340-98505fe6475d-loader
2023-10-03T07:45:39.979797Z [info     ] On instance['web_rec_start_date']: cmd_type=elb consumer=True name=table-specifier-cede371e-e0db-41b5-8340-98505fe6475d-loader producer=False stdio=stderr string_id=table-specifier-cede371e-e0db-41b5-8340-98505fe6475d-loader
2023-10-03T07:45:39.979846Z [info     ]     '1997-08-16T00:00:00+00:00' cmd_type=elb consumer=True name=table-specifier-cede371e-e0db-41b5-8340-98505fe6475d-loader producer=False stdio=stderr string_id=table-specifier-cede371e-e0db-41b5-8340-98505fe6475d-loader
2023-10-03T07:45:41.736450Z [error    ] Loader failed
2023-10-03T07:45:41.736638Z [error    ] Block run completed.
u
@nidhi_kakulawaram this looks similar to https://github.com/meltano/sdk/issues/1831
n
Yes, it is! Seems like it is a known issue that you all are trying to address with singer tap directly. How should I work around for the time being while waiting for the fix for this bug to go out? would you suggest a tap-snowflake fork with something like this? https://github.com/MeltanoLabs/tap-postgres/pull/172/files
I opened up a PR for tap-snowflake so the conversion from date to datetime doesnt occur unnecessarily: https://github.com/MeltanoLabs/tap-snowflake/pull/34