a question about deploying Meltano for a low-cost, stateless deployment: https://docs.meltano.com/concepts/state_backends/ says

for Meltano projects running in ephemeral environments or in circumstances where administering a dedicated backend database is undesirable, Meltano also supports persisting state in remote state backends backed by cloud storage services

... this looks ideal to me, and in fact, I have a working Docker build prepped and ready to go; however I'm wondering: are people deploying in this fashion simply not taking advantage of catalog caching? Are you caching the schema/catalog as part of your container build?

Personally, I have never made use of catalog caching when using Meltano BUT that is not because it isn't a useful feature. The reality is for my use case, usually the tap has the schema for the streams built in so the caching isn't really something that I care much about. They are pretty small so I'd spend more time building a process to pre-cache the catalog than I would save. (example: https://github.com/MeltanoLabs/tap-github/blob/main/tap_github/organization_streams.py) However, it definitely would be useful for going against a well structured database in which defining the schema directly into the tap doesn't make as much sense. My problem was that Oracle (🤮) absolutely despises doing things in a sensible way so it was too difficult to actually use discovery to get the schema. There were simply too many prebuilt objects (10,000+ in a single schema) and the discovery took forever unless I put in some pretty rigid select filters. I figured it was just easier to just mount a directory of catalog JSON files to the docker container that users can add and remove from freely.

Thanks, that is helpful. I'm pulling from 1) a PostgreSQL DB, which has a clean schema but I'm also only syncing 5 tables out of several hundred so catalog refresh is slow (though not as bad as your Oracle example 😉 ), and 2) several DynamoDB tables, whose catalog autodetection (based on sample records) was insufficient and needed several extractor-config "schema" overrides. For the second, I already was considering switching to static catalogs ("what happens if the sample-record set changes during runtime, and my overrides need more rules to unbreak it"). Anyhow, perhaps static catalogs would be a good solution for both.

I am using AWS Batch for ephemeral Meltano runs. I'm using S3 for the state backend but then I have mounted EFS on the containers where I cache the catalogs. I did have to build another layer of caching in to grab that tap's catalog before the run and put a copy back after the run. It's been a while but I think I had to do this because I didn't want the entire

.meltano

folder shared between all runs and I run a job for every stream of the tap separately

The super simple way to use a catalog cache if you want to is spin up a VM and just run meltano on that machine via cron / gitlab CI etc and it would all just work for "free" For most use cases I see with meltano this honestly is the best solution as it's so simple