❯ meltano --log-level=debug run tap-rest-api-msdk target-jsonl

[...]

2023-10-23T15:21:02.197273Z [info     ] Exception: Message is missing required key 'key_properties': {'type': 'SCHEMA', 'stream': 'query 1', 'schema': {'properties': {'posts': {'items': {'properties': {'thread': {'properties': {'uuid': {'type': 'string'}, 'url': {'type': 'string'}, 'site_full': {'type': 'string'}, 'site': {'type': 'string'}, 'site_section': {'type': 'string'}, 'site_categories': {'items': {'type': 'string'}, 'type': 'array'}, 'section_title': {'type': 'string'}, 'title': {'type': 'string'}, 'title_full': {'type': 'string'}, 'published': {'type': 'string'}, 'replies_count': {'type': 'number'}, 'participants_count': {'type': 'number'}, 'site_type': {'type': 'string'}, 'country': {'type': 'string'}, 'main_image': {'type': 'string'}, 'performance_score': {'type': 'number'}, 'domain_rank': {'type': 'number'}, 'domain_rank_updated': {'type': 'string'}, 'reach': {}, 'social': {'properties': {'facebook': {'properties': {'likes': {'type': 'number'}, 'comments': {'type': 'number'}, 'shares': {'type': 'number'}}, 'type': 'object'}, 'gplus': {'properties': {'shares': {'type': 'number'}}, 'type': 'object'}, 'pinterest': {'properties': {'shares': {'type': 'number'}}, 'type': 'object'}, 'linkedin': {'properties': {'shares': {'type': 'number'}}, 'type': 'object'}, 'stumbledupon': {'properties': {'shares': {'type': 'number'}}, 'type': 'object'}, 'vk': {'properties': {'shares': {'type': 'number'}}, 'type': 'object'}}, 'type': 'object'}}, 'type': 'object'}, 'uuid': {'type': 'string'}, 'url': {'type': 'string'}, 'ord_in_thread': {'type': 'number'}, 'parent_url': {}, 'author': {'type': 'string'}, 'published': {'type': 'string'}, 'title': {'type': 'string'}, 'text': {'type': 'string'}, 'highlightText': {'type': 'string'}, 'highlightTitle': {'type': 'string'}, 'highlightThreadTitle': {'type': 'string'}, 'language': {'type': 'string'}, 'external_links': {'items': {}, 'type': 'array'}, 'external_images': {'items': {}, 'type': 'array'}, 'entities': {'properties': {'persons': {'items': {}, 'type': 'array'}, 'organizations': {'items': {}, 'type': 'array'}, 'locations': {'items': {'properties': {'name': {'type': 'string'}, 'sentiment': {'type': 'string'}}, 'type': 'object'}, 'type': 'array'}}, 'type': 'object'}, 'rating': {}, 'crawled': {'type': 'string'}, 'updated': {'type': 'string'}}, 'type': 'object'}, 'type': 'array'}}, 'type': 'object'}} cmd_type=elb consumer=True name=target-jsonl producer=False stdio=stderr string_id=target-jsonl

I've searched Slack history, but only found this https://meltano.slack.com/archives/C013X86PGUS/p1653768274515049

Does it have anything to do with

primary_keys

? I assume this is correct`primary_keys: ["uuid"]` if

"records_path": "$.posts[*]"

and http response is

{"posts": [{"uuid": "123", ...}, {"uuid": "abcd", ...}]}

either way, I tried with both

primary_keys: ["uuid"]

and

primary_keys: ["posts.uuid"]

and same error 🤔

ok, had to look a bit deeper and realised

key_properties

is an actual mandatory field that you have to define in the `schema.json`https://github.com/singer-io/singer-python/blob/master/singer/messages.py#L208