Hi everyone, i'm trying to implement state for one of my REST streams because it supports "updatedAt". I am plaining to pull the data from multiple accounts, using environment variables to pass the required information and i would like to save the state for each account. The account name it is not a paramater for the API. I tried by using state partitions, passing the account name into the context, but every time the stream runs with a new account it does not create a new state for that account but it replaces the last one. So below an example of what the state for the stream looks like:

{"drivers": {"partitions": [{"context": {"fleet": "ABC"}, "replication_key": "updatedAtTime", "replication_key_value": "2024-05-07T11:00:24.769085Z"}]}

Is it possible to parse the state for the current fleet, and only update that one after the job is finished? Many thanks in advance 🙂

Depending on the number of accounts you have, an inherited tap for each account might be one approach. So when you run

tap-myapi-env2 target-postgres

it doesn't clobber the state for

tap-myapi-env1 target-postgres

Scaling is the problem, i do not want to create a new tap for each account because it is going to get really big quick. I find it much easier to work with one tap and manage the runtime using env variables. It is an awsome option from meltano. Been reading the sdk for a while and some other rest taps from the hub but did not find anything usefull

Figures, interested to know if you get a solution.

I will let you know 🙂

@Eni S which state backend are you using? With something like S3, you could configure the state_backend.uri to include an env var that expands with the account ID.

I am using S3 and with Env variables. Great Tip, thank you very much @Edgar Ramírez (Arch.dev) 🙏

Love the solution here, I need to keep that one in the back pocket.