#6154 Meltano could warn if `replication-key` metadata is set to property not set in table `SCHEMA` Issue created by astrojuanlu When setting

replication-method: INCREMENTAL

and the wrong

replication-key

, the following error is raised:

$ meltano invoke tap-postgres
2022-06-10T15:04:26.945653Z [info     ] Environment 'dev' is active
time=2022-06-10 17:04:29 name=tap_postgres level=INFO message=Selected streams: ['public-countries', 'public-departments', 'public-dependents', 'public-employees', 'public-jobs', 'public-locations', 'public-regions'] 
time=2022-06-10 17:04:29 name=tap_postgres level=INFO message=No streams marked as currently_syncing in state file
time=2022-06-10 17:04:29 name=tap_postgres level=INFO message=Beginning sync of stream(public-countries) with sync method(incremental)
time=2022-06-10 17:04:29 name=tap_postgres level=INFO message=Stream public-countries is using incremental replication with replication key id
{"type": "SCHEMA", "stream": "public-countries", "schema": {"type": "object", "properties": {"country_id": {"type": ["string"], "maxLength": 2}, "country_name": {"type": ["null", "string"], "maxLength": 40}, "region_id": {"type": ["null", "integer"], "minimum": -2147483648, "maximum": 2147483647}}, "definitions": {"sdc_recursive_integer_array": {"type": ["null", "integer", "array"], "items": {"$ref": "#/definitions/sdc_recursive_integer_array"}}, "sdc_recursive_number_array": {"type": ["null", "number", "array"], "items": {"$ref": "#/definitions/sdc_recursive_number_array"}}, "sdc_recursive_string_array": {"type": ["null", "string", "array"], "items": {"$ref": "#/definitions/sdc_recursive_string_array"}}, "sdc_recursive_boolean_array": {"type": ["null", "boolean", "array"], "items": {"$ref": "#/definitions/sdc_recursive_boolean_array"}}, "sdc_recursive_timestamp_array": {"type": ["null", "string", "array"], "format": "date-time", "items": {"$ref": "#/definitions/sdc_recursive_timestamp_array"}}, "sdc_recursive_object_array": {"type": ["null", "object", "array"], "items": {"$ref": "#/definitions/sdc_recursive_object_array"}}}}, "key_properties": ["country_id"], "bookmark_properties": ["id"]}
{"type": "STATE", "value": {"bookmarks": {"public-countries": {"last_replication_method": "INCREMENTAL", "replication_key": "id", "version": 1654873469915}, "public-departments": {"last_replication_method": "INCREMENTAL"}, "public-dependents": {"last_replication_method": "INCREMENTAL"}, "public-employees": {"last_replication_method": "INCREMENTAL"}, "public-jobs": {"last_replication_method": "INCREMENTAL"}, "public-locations": {"last_replication_method": "INCREMENTAL"}, "public-regions": {"last_replication_method": "INCREMENTAL"}}, "currently_syncing": "public-countries"}}
{"type": "ACTIVATE_VERSION", "stream": "public-countries", "version": 1654873469915}
time=2022-06-10 17:04:29 name=tap_postgres level=CRITICAL message='NoneType' object has no attribute 'get'
Traceback (most recent call last):
  File "/home/juanlu/Projects/Orchest/Singer/test-replication-none/.meltano/extractors/tap-postgres/venv/bin/tap-postgres", line 8, in <module>
    sys.exit(main())
  File "/home/juanlu/Projects/Orchest/Singer/test-replication-none/.meltano/extractors/tap-postgres/venv/lib/python3.8/site-packages/tap_postgres/__init__.py", line 435, in main
    raise exc
  File "/home/juanlu/Projects/Orchest/Singer/test-replication-none/.meltano/extractors/tap-postgres/venv/lib/python3.8/site-packages/tap_postgres/__init__.py", line 432, in main
    main_impl()
  File "/home/juanlu/Projects/Orchest/Singer/test-replication-none/.meltano/extractors/tap-postgres/venv/lib/python3.8/site-packages/tap_postgres/__init__.py", line 421, in main_impl
    do_sync(conn_config, args.catalog.to_dict() if args.catalog else args.properties,
  File "/home/juanlu/Projects/Orchest/Singer/test-replication-none/.meltano/extractors/tap-postgres/venv/lib/python3.8/site-packages/tap_postgres/__init__.py", line 312, in do_sync
    state = sync_traditional_stream(conn_config,
  File "/home/juanlu/Projects/Orchest/Singer/test-replication-none/.meltano/extractors/tap-postgres/venv/lib/python3.8/site-packages/tap_postgres/__init__.py", line 175, in sync_traditional_stream
    state = do_sync_incremental(conn_config, stream, state, desired_columns, md_map)
  File "/home/juanlu/Projects/Orchest/Singer/test-replication-none/.meltano/extractors/tap-postgres/venv/lib/python3.8/site-packages/tap_postgres/__init__.py", line 86, in do_sync_incremental
    state = incremental.sync_table(conn_config, stream, state, desired_columns, md_map)
  File "/home/juanlu/Projects/Orchest/Singer/test-replication-none/.meltano/extractors/tap-postgres/venv/lib/python3.8/site-packages/tap_postgres/sync_strategies/incremental.py", line 61, in sync_table
    replication_key_sql_datatype = md_map.get(('properties', replication_key)).get('sql-datatype')
AttributeError: 'NoneType' object has no attribute 'get'

Fixing the

replication-key

solves the problem: $ diff -u meltano_bad.yml meltano.yml --- meltano_bad.yml 2022-06-10 173920.482379726 +0200 +++ meltano.yml 2022-06-10 173934.874301569 +0200 @@ -10,7 +10,7 @@ metadata: '*': replication-method: INCREMENTAL - replication-key: id + replication-key: region_id loaders: - name: target-postgres variant: transferwise meltano/meltano