Hi team is there a way to override stream attribute type wit Meltano #troubleshooting

Hi team, is there a way to override stream attribu...

rustam_a

11/03/2023, 4:48 PM

Hi team, is there a way to override stream attribute type without changing source code of the tap? In my case, I'm using

tap-braintree

through Airbyte, and getting the following error when trying to load payment transactions:

Copy code

ValueError: No format in ['%Y-%m-%d %H:%M:%S'] matching 2023-10-03

rustam_a

11/03/2023, 5:03 PM

Full stack trace:

Copy code

2 validation errors for Transaction
subscription_details -> billing_period_end_date
  invalid type; expected datetime, string, bytes, int or float (type=type_error)
subscription_details -> billing_period_start_date
  invalid type; expected datetime, string, bytes, int or float (type=type_error)
Traceback (most recent call last):
  File "/airbyte/integration_code/main.py", line 13, in <module>
    launch(source, sys.argv[1:])
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 200, in launch
    for message in source_entrypoint.run(parsed_args):
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 114, in run
    yield from map(AirbyteEntrypoint.airbyte_message_to_string, self.read(source_spec, config, config_catalog, state))
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 156, in read
    yield from self.source.read(self.logger, config, catalog, state)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/manifest_declarative_source.py", line 136, in read
    yield from super().read(logger, config, catalog, state)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 142, in read
    raise e
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 122, in read
    yield from self._read_stream(
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 197, in _read_stream
    for record in record_iterator:
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 328, in _read_full_refresh
    for record_data_or_message in record_data_or_messages:
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/declarative_stream.py", line 104, in read_records
    yield from self.retriever.read_records(stream_slice)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 304, in read_records
    for stream_data in self._read_pages(self._parse_records, self.state, stream_slice):
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 283, in _read_pages
    yield from records_generator_fn(response, stream_state, stream_slice)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 366, in _parse_records
    yield from self._parse_response(response, stream_slice=stream_slice, stream_state=stream_state)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 227, in _parse_response
    records = self.record_selector.select_records(
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/extractors/record_selector.py", line 44, in select_records
    all_data = self.extractor.extract_records(response)
  File "/airbyte/integration_code/source_braintree/source.py", line 126, in extract_records
    return [
  File "/airbyte/integration_code/source_braintree/source.py", line 127, in <listcomp>
    Transaction(**self._get_json_from_resource(BTransaction(None, transaction))).dict(exclude_unset=True)
  File "/usr/local/lib/python3.9/site-packages/pydantic/main.py", line 341, in __init__
    raise validation_error
pydantic.error_wrappers.ValidationError: 2 validation errors for Transaction
subscription_details -> billing_period_end_date
  invalid type; expected datetime, string, bytes, int or float (type=type_error)
subscription_details -> billing_period_start_date
  invalid type; expected datetime, string, bytes, int or float (type=type_error)'}

Open in Slack

Previous Next