Hey all, I’m testing out developing my new tap using the Meltano SDK using the REST base. I can see how much faster one can develop taps once they get the hang of it. As I’ve mentioned on a previous office hours session, I have to make N calls to the API for N domains (passed in through config), e.g. config:

{ "domains": ["<http://shopify.com|shopify.com>", "<http://meltano.com|meltano.com>", ...]

I’m calling the endpoint with each domain e.g.

api/{domain}/data

. I need to be able to track state for each domain:

{
   "mystream": {
       "<http://shopify.com|shopify.com>": { "status": "success", "status_dts": "[UTC_dt]"},
       "<http://meltano.com|meltano.com>": { "status": "entity_not_found", "status_dts": "[UTC_dt]"},
        ...
    }

}

Where’s the best place to manage the state for this kind of thing? In every example I read, they are using a simple bookmark, but I need fine-grain control.

Apologies if we already talked through this option but have you looked into the partitions feature? If you can populate the partitions list upfront, that would automatically create individual states per partition.

I can. I thought that might be useful in some way, but I couldn’t make heads or tails of it through the docs without an example

Understood. Sorry we don't have a lot of examples on this yet. Here's one example which might help... https://gitlab.com/meltano/sdk/-/blob/main/singer_sdk/samples/sample_tap_gitlab/gitlab_rest_streams.py#L68

perfect that’s very helpful