Hey all, happy good friday: I believe I have foun...
# troubleshooting
s
Hey all, happy good friday: I believe I have found a bug (or witchcraft), and would appreciate feedback before submitting a proper issue. I'm currently creating a hubspot tap using the meltano sdk, and when trying to rerun a tap, Im getting a key error on my replication key. I investigated, and I can see my key in the records, but it suddenly gets written off before being called. Here is the code from core.py
Copy code
for record_result in self.get_records(current_context):
                <http://logging.info|logging.info>('sb - RECORDS')
                <http://logging.info|logging.info>(record_result)
                if isinstance(record_result, tuple):
                    # Tuple items should be the record and the child context
                    record, child_context = record_result
                else:
                    record = record_result
                child_context = copy.copy(
                    self.get_child_context(record=record, context=child_context)
                )
                for key, val in (state_partition_context or {}).items():
                    # Add state context to records if not already present
                    if key not in record:
                        record[key] = val
                <http://logging.info|logging.info>('sb - RECORDS_2')
                <http://logging.info|logging.info>(record)
                # Sync children, except when primary mapper filters out the record
                if self.stream_maps[0].get_filter_result(record):
                    self._sync_children(child_context)
                self._check_max_record_limit(record_count)
                <http://logging.info|logging.info>('sb - RECORDS_3')
                <http://logging.info|logging.info>(record)
                self._write_record_message(record)
                <http://logging.info|logging.info>('sb - RECORDS_4')
                <http://logging.info|logging.info>(record)
                if selected:
                    if (record_count - 1) % self.STATE_MSG_FREQUENCY == 0:
                        self._write_state_message()
                    self._write_record_message(record)
                    <http://logging.info|logging.info>('sb - RECORDS_5')
                    <http://logging.info|logging.info>(record)
                    try:
                        <http://logging.info|logging.info>('sb - RECORDS_6')
                        <http://logging.info|logging.info>(record)
                        self._increment_stream_state(record, context=current_context)
And the subsequent output: ```time=2022-04-15 155903 name=tap-hubspot level=INFO message=Tap has custom mapper. Using 1 provided map(s). {"type": "SCHEMA", "stream": "companies", "schema": {"properties": {"id": {"type": ["string", "null"]}}, "type": "object"}, "key_properties": ["id"], "bookmark_properties": ["updatedAt"]} time=2022-04-15 155905 name=tap-hubspot level=INFO message=INFO METRIC: {'type': 'timer', 'metric': 'http_request_duration', 'value': 2.352536, 'tags': {'endpoint': '/crm/v3/objects/companies', 'http_status_code': 200, 'status': 'succeeded'}} time=2022-04-15 155905 name=root level=INFO message=sb - RECORDS time=2022-04-15 155905 name=root level=INFO message={'id': '262370061', 'properties': {'createdate': '2017-01-24T224405.677Z', 'domain': 'falafelavenue.com', 'hs_lastmodifieddate': '2022-04-08T151156.740Z', 'hs_object_id': '262370061', 'name': 'Falafel Avenue'}, 'createdAt': '2017-01-24T224405.677Z', 'updatedAt': '2022-04-08T151156.740Z', 'archived': False} time=2022-04-15 155905 name=root level=INFO message=sb - RECORDS_2 time=2022-04-15 155905 name=root level=INFO message={'id': '262370061', 'properties': {'createdate': '2017-01-24T224405.677Z', 'domain': 'falafelavenue.com', 'hs_lastmodifieddate': '2022-04-08T151156.740Z', 'hs_object_id': '262370061', 'name': 'Falafel Avenue'}, 'createdAt': '2017-01-24T224405.677Z', 'updatedAt': '2022-04…
# TODO: Make issue or remove element replication_value = None if self.replication_key in record_result: replication_value = record_result[self.replication_key]
My super duper fix
e
Is the replication key included in the stream schema?
I left a comment in the issue 😄
s
Thanks! I answered there. For those interested, here is a link to the issue: https://gitlab.com/meltano/meltano/-/issues/3404
Yeah, I'm fairly confident this was an issue brought on by my stupidity. Closing it for now, hoping to never open it again 😉