Hello! Something @edgar_ramirez_mondragon mentioned about the meltano state database just being a key-value store made me consider if we could use AWS Systems Manager Parameter Store (SSM) as a state backend (and there are likely similar services in GCS/Azure): Pros (admittedly a few things I'm unsure about here): - The SSM API is built for retrieving parameters by name so it might be more performant than S3 - Versioning is baked in by default - It may be cheaper Cons: - Parameter names don't accept ':' so I replaced those with '/' making the environment, tap/target, state-suffix-id parts of the path - You might be storing other stuff in the SSM Parameter store and don't want to clutter it up with state values (but you can use paths as namespaces to organise things) Anyway, despite the cons, I decided to throw together something on a feature branch https://github.com/mjsqu/meltano/tree/feat/aws_ssm_state_backend - you just specify:

MELTANO_STATE_BACKEND_URI='<ssm-parameter://meltano/state_test>'

and ensure the relevant AWS environment variables are set, and you should be good to go. If nothing else, it was fun and surprisingly easy to add my own custom state backend to Meltano because of how well it's written (like

meltano state copy

just worked once I had set() and get() in place! 😎_)_, so even if this doesn't have any mileage, I may write a documentation page in the contributing section about how it is possible.

Hey @mark_johnston! Yeah, ideally we'd want to support plugins (surely with a different name, to avoid confusion 😅). That way folks can develop and maintain them on their own if they're interested in using a niche service as a state backend. Probably only a mild refactoring is required to support that.

Migrated to https://github.com/meltano/meltano/discussions/8276