I've been looking at migrating out of a postgresql backend to an s3 backend. It would simplify our infrastructure as we wouldn't have to worry about a database for meltano. wondering if folks know of any limitations with the file based state backend vs a database. for some additional context, our meltano jobs are containerized and will run concurrently. e.g. instead of defining a

tap-salesforce

with a list of all select rules, we have something like below, and then the docker containers will concurrently fetch data from all the salesforce streams.

plugins:
  extractors:
    - name: tap-salesforce
      config:
        ...
    - name: tap-salesforce-accounts
      inherit_from: tap-salesforce
      select:
        - Account.Id
        ...

    - name: tap-salesforce-opportunities
      inherit_from: tap-salesforce
      select:
        - Opportunity.Id
        ...

I have some fear (likely unfounded) that these concurrently running containers will try to edit the same file simultaneously, and I'm hoping to learn more so I can decide whether the s3 based backend would be appropriate for how I use meltano.

I have the same fear 😄 , there was work done on locking though so I'd read through https://docs.meltano.com/concepts/state_backends/#locking from a quick skim this looks pretty slick to me!

https://github.com/meltano/meltano/issues/8435 may be relevant here. > I've been looking at migrating out of a postgresql backend to an s3 backend. It would simplify our infrastructure as we wouldn't have to worry about a database for meltano. We are doing the same - one thing I have learned is that moving from “Postgres system database” to “s3 state backend” is that in the latter, only state is persisted - run data is not. That may or may not be a concern. Also, meltano state records are distinct per tap - your “tap-salesforce-accounts” and “tap-salesforce-opportunities” taps will use different s3 objects for state.