Hi, does anyone is using Meltano for managing the some more complex data engineering needs? Let’s say we have > 100 sources from different sources (s3/databases/API), hundreds of tables, hundreds of daily/hourly jobs? Do you have some articles about such experiences?

I’m not aware of any articles for such large scales - but I’d love to see some!

Ya know, just a random thought, but as more medium-to-large-scale deployments of Meltano start to emerge, it could be kinda cool to gradually grow a collection of case studies.

For sure - we just need some more people 😅

Of course. I’m just saying as the community continues to grow and people write up articles about their experiences, it could be cool if they contributed them to a section under the website or something like that.

Hey @andrew_stewart we don't have exactly the setup you are talking about, but we do have many streams across a growing number of sources. Meltano has some newer functionality that allows us to be specific about the sources, streams, and fields we want any job to sync. We have multiple jobs across multiple clients. Already we are seeing the value in combining the Meltano selection features with our orchestrator (Prefect). It is scaling nicely. I don't have any articles on it, but it looks like it will be manageable as we scale