Hello! I'm working on setting up a simple poc to showcase Meltano to my colleagues. I was aiming for fetching some data off our Kafka and then store it in our S3. However our Kafka cluster is extremely old (v 0.10 to be exact) so some of our messages on Kafka does not have a Kafka timestamp so I keep getting the following error.

WARNING Unknown message type ACTIVATE_VERSION in message {'type': 'ACTIVATE_VERSION', 'stream': 'ingest', 'version': 1696426379397}
time=2023-10-04 15:33:04 name=tap_kafka level=CRITICAL message=Required timestamp not available in the kafka message. cmd_type=elb consumer=False name=tap-kafka producer=True stdio=stderr string_id=tap-kafka

Is there any way to get around this? I tried excluding the

message_timestamp

attribute but that didn't seem to help.

Hi Tim 👋🏼. The message timestamp seems to be used in two places: to generate the incremental replication bookmark and as the

message_timestamp

field in every record. It doesn't seem possible to avoid that error by deselecting the field, since it's computed regardless. It also doesn't seem possible to disable the computation for bookmarks. Your only option would be to fork the tap to change the behavior, although I'm not familiar with Kafka or the tap to know how to handle the missing timestamps and whether you'd risk data loss if you simply skip incrementing state whenever they're unavailable.