Hi, I'm new to Meltano and data engineering in general. I'm working on a project that requires me extracting info from using REST, and has start_id and end_id as params. I've got a custom python script that extracts the maximum id from a database, and uses that to populate the start_id. As I don't know the maximum id from the source I'm extracting from, I'm just adding 100 to this id for the end_id to load in chunks of JSON until the response is empty. I'm assuming my options are 1. try and get the api updated to use the settings the existing REST extractor uses (I noticed pagination) 2. write a custom extractor but would this work for my scenario? 3. continue to use my script to load the info, and just use Meltano for the load part. Sorry for the wall of text! I'm just starting with Meltano as a POC so looking for some direction, thanks.

2. write a custom extractor but would this work for my scenario?

This is definitely doable. The SDK is pretty flexible these days, especially with pagination support.

3. continue to use my script to load the info, and just use Meltano for the load part.

This is possible, but your script would have to output Singer messages to load with a target using Meltano. The SDK already abstracts away this implementation detail (and more): https://sdk.meltano.com/en/latest/implementation/index.html# Personally, I think you would benefit from creating a custom tap using the SDK - especially if

tap-rest-api-msdk

doesn't currently support the kind of pagination logic you described. It sounds like the problem you are trying to solve with your custom script is exactly what the SDK was designed for! 😅

Thanks Reuben, appreciate the advice. Are there are samples of REST custom taps that the community have created maybe?

Shameless plug, but here are a couple of ours: https://github.com/Matatika/tap-shopify https://github.com/Matatika/tap-veeqo https://github.com/Matatika/tap-auth0 https://github.com/Matatika/tap-trello https://github.com/Matatika/tap-spotify You can also checkout Meltano Hub - any plugins built with the SDK have a

Built with the Meltano SDK

badge (all of the most popular currently are).