Hi Folks, a quick note to say there has been a big update to tap-rest-api-msdk, it has many new features to allow you to connect into more API's via configuration without the need to write a specific tap. At the bottom of the README.md there are examples with config for a few public API's. Latest version released is 1.3.0. I am keen to hear if there are any issue with my PR to this tap. This PR introduces new Authenticators and Paginators to

tap-rest-api-msdk

- making this tap the swiss army knife for accessing API's. Summary - Support for most Meltano SDK Authenticators. - Support for all Meltano Paginators. - Flexibility to support many new API's by new settings to adjust request parameter names. See README.md for more details on settings. - Ability to send parameters in the request body rather than request parameters (if required). - Moving from deprecated

get_next_page_token

to support

get_new_paginator

. This removes the warnings in the logs. - Enhanced incremental replication (include support for API query templates). - New modules

auth

and

pagination

keeping a clean design. - New

auth

method aws, to support ingestion from AWS REST End-point e.g. OpenSearch. Paginators Each REST API is different. This PR builds on the concept of picking an appropriate request and response style for the API. Select an appropriate

pagination_request_style

to pick the paginator you require. In most cases this needs to be coupled with an appropriate

paginator_response_style

to process the response and pick the next page location in the body or headers. Supported Paginators as part of this PR include: -

jsonpath_paginator

or

default

- This style obtains the token for the next page from a specific location in the response body via JSONPath notation. In many situations the

jsonpath_paginator

is a more appropriate paginator to the

hateoas_paginator

. -

next_page_token_path

- The jsonpath to next page token. Example:

"$['@odata.nextLink']"

, this locates the token returned via the Microsoft Graph API. Default

'$.next_page'

for the

jsonpath_paginator

paginator only otherwise None. -

offset_paginator

or

style1

- This style uses URL parameters named offset and limit -

offset

is calculated from the previous response, or not set if there is no previous response -

pagination_page_size

- Sets a limit to number of records per page / response. Default

25

records. -

pagination_limit_per_page_param

- the name of the API parameter to limit number of records per page. Default parameter name

limit

. -

pagination_total_limit_param

- The name of the param that indicates the total limit e.g. total, count. Defaults to total -

next_page_token_path

- Used to locate an appropriate link in the response. Default None - but looks in the

pagination

section of the JSON response by default. Example, jsonpath to get the offset from the NOAA API

'$.metadata.resultset'

. -

simple_header_paginator

- This style uses links in the Header Response to locate the next page. Example the

x-next-page

link used by the Gitlab API. -

header_link_paginator

- This style uses the default header link paginator from the Meltano SDK. -

restapi_header_link_paginator

- This style is a variant on the header_link_paginator. It supports the ability to read from GitHub API. -

pagination_page_size

- Sets a limit to number of records per page / response. Default

25

records. -

pagination_limit_per_page_param

- the name of the API parameter to limit number of records per page. Default parameter name

per_page

. -

pagination_results_limit

- Restricts the total number of records returned from the API. Default None i.e. no limit. -

hateoas_paginator

- This style parses the next_token response for the parameters to pass. It is used by API's utilising the HATEOAS Rest style <https://en.wikipedia.org/wiki…