<!channel> Hey Meltano community, as we’ve teased a couple of times over the past few months, we’ve been working on something new that we’re super excited to share with you today: arch Arch is our new product that brings powerful data engineering capabilities to software teams building products on top of their customers’ data, so that they can focus on using the data rather than the data engineering involved in acquiring it. Arch is built on top of melty Meltano, so you can rest assured that the open source project and community that you know and love aren’t going anywhere 🙂 I’ve written more about the change in our company’s focus and how it relates to Meltano and Meltano Cloud on the Meltano blog, and on the new Arch blog we’ve got posts up about the motivation and long-term vision and the product itself. If you’ve been trying to use Meltano directly but Arch sounds like a better fit, we’d love to work with you — there are links on the Arch website to book a slot on my calendar, get access to Arch, or just stay in the loop!

Wow, congrats! I need to read all of these in details 🙂

Congratz! Question: should Try live demo link on this page: https://meltano.com/blog/were-bringing-powerful-data-engineering-capabilities-to-software-teams-with-arch/ Lead to this Github repo? https://github.com/meltano/meltano-codespace-ready

@jan_soubusta Ah very good call, that’s a Meltano blog element that doesn’t take into account the post itself 😅 Lemme see if we can fix that

Question2: If I understand the intro video well, the final result of Arch is a data model in PostgreSQL database. If yes, I could create a demo integrating our analytics platform with the result of Arch project. Would you be interested in collaborating on something like that? Regarding embeddings - this is interesting. Immediately an idea appeared in my head - all BI/analytics filters could utilize the embeddings, so instead of exact match/like we could find synonyms, etc.

@jan_soubusta

If I understand the intro video well, the final result of Arch is a data model in PostgreSQL database.

That’s right, you can access the raw and modeled data directly over SQL, or define a REST/GraphQL API over it

If yes, I could create a demo integrating our analytics platform with the result of Arch project.

Would you be interested in collaborating on something like that?

Definitely! Wanna book a slot on arch.dev/douwe to catch up and discuss?

Regarding embeddings - this is interesting. Immediately an idea appeared in my head - all BI/analytics filters could utilize the embeddings, so instead of exact match/like we could find synonyms, etc.

Exactly, Arch can automatically generate embeddings for any text data, and you can do similarity search against them for natural-language search, fuzzy matching, or LLM RAG

Will contact you once my hands are more free. I have to finish an integration with MotherDuck first 😉

Amazing guys! How does the multi-tenancy work? Also, the pricing of $0.75 per compute hour is not very clear in terms of storage, multiple tenants etc? Does this sit on it's own dedicated VM somewhere with the stack installed, or does Arch have a farm of machines which does the syncing etc and you get charged for what you use? Is data at rest charged as well?

@johann_du_toit Thanks Johann!

How does the multi-tenancy work?

In Arch it means that you can define sources, mappings, and data models (effectively ELT pipelines, but declaratively defined) that can then have multiple tenants under them, representing credentials possibly onboarded using OAuth. Each tenant gets its own pipelines run, and the data can be looked up segmented by tenant, and stored in separate per-tenant schemas or a multi-tenant table with a tenant_id, as appropriate.

Also, the pricing of $0.75 per compute hour is not very clear in terms of storage, multiple tenants etc?

It’s the total compute resources used across all tenants, across the EL, T, and querying (SQL/API) steps. Right now now we’re not factoring in storage volume explicitly as compute usage and storage are correlated, but there may be limits (with additional pricing past some storage volume) in the future.

Does this sit on it’s own dedicated VM somewhere with the stack installed, or does Arch have a farm of machines which does the syncing etc and you get charged for what you use?

The architecture is multi-tenant on shared (isolated) compute resources, so not dedicated VMs and you only pay for what you use.

Is data at rest charged as well?

As of now, no. But I could see us intelligently moving rarely used data to cheaper/higher-latency storage for example to make that work (with an ability to opt out of that by paying for storage past a certain point). Let’s catch up soon — feel free to book some time on arch.dev/douwe!

Thanks Douwe. I think similar to Jan, despite your focus on AI, this is definitely what analytics SaaS products like us were looking for. Another question on the billing... would it be possible to get the billing broken down per tenant?

I think similar to Jan, despite your focus on AI, this is definitely what analytics SaaS products like us were looking for.

That’s great to hear! It’s definitely not exclusively for AI, but there’s a big opportunity to be better there than any of the existing “embedded EL” style options, and it’s just a matter of time until every SaaS product does at least some AI!

Another question on the billing... would it be possible to get the billing broken down per tenant?

For sure, I should’ve specified that

Congrats on the launch! What are you using for columnar postgres? I’m sure you’ve already seen it, but https://www.hydra.so/ looks pretty cool.

@mike_ritchie Thanks! We’re working with Tembo on the Postgres side, who are indeed using hydra_columnar: https://tembo.io/docs/tembo-stacks/olap

That's even more interesting now. Columnar is always much better for analytics 😉