In <tap-rest-api-msdk> there is a section in the t...
# singer-tap-development
s
In tap-rest-api-msdk there is a section in the tap to dynamically discover a schema based on the data returned from the API - https://github.com/Widen/tap-rest-api-msdk/blob/4f87c1adae00446388ebbe418c70b87c231856dc/tap_rest_api_msdk/tap.py#L532-L609. If the data returned by the API is consistent, i.e. the same number of attributes in every record returned, the schema dynamic discovery works well. If however the API has certain attributes which are optional, and therefore only appear in certain records it can lead to data loss. Specifically I have seen where the random sampling of a certain number of records to determine the schema may miss certain attributes. When the tap is emitting records, Meltano will ignore any attributes that weren't part of the original schema. Now one way to solve this is to supply a schema manually and not do the automatic discovery. The problem is if a new attribute is added to the API this will result in the same problem if this is missed. So I am wondering is there a way if a new attribute is discovered to be missing in the schema whether it could be handled gracefully e.g. perhaps a further schema message is issued to alter the table (I am assuming a database target - snowflake) so that the attribute is not lost? Keen to get some thoughts on this as I really love the dynamic discovery. For now I am emitting the raw json record and unpacking it in dbt, but thought there must be a better way.