https://linen.dev logo
#singer-tap-development
Title
# singer-tap-development
b

borna_almasi

07/10/2021, 2:47 AM
Hey all, I’m testing out developing my new tap using the Meltano SDK using the REST base. I can see how much faster one can develop taps once they get the hang of it. As I’ve mentioned on a previous office hours session, I have to make N calls to the API for N domains (passed in through config), e.g. config:
Copy code
{ "domains": ["<http://shopify.com|shopify.com>", "<http://meltano.com|meltano.com>", ...]
I’m calling the endpoint with each domain e.g.
api/{domain}/data
. I need to be able to track state for each domain:
Copy code
{
   "mystream": {
       "<http://shopify.com|shopify.com>": { "status": "success", "status_dts": "[UTC_dt]"},
       "<http://meltano.com|meltano.com>": { "status": "entity_not_found", "status_dts": "[UTC_dt]"},
        ...
    }

}
Where’s the best place to manage the state for this kind of thing? In every example I read, they are using a simple bookmark, but I need fine-grain control.