Hello Meltano community!
I'm curious if you know any any case studies/success stories of using Meltano in teams with limited tech experience (especially in conjuction with more technically challenging tools like Airflow)? I believe there were personas like this were considered at least a year ago, but it's not clear to me how Meltano has changed since.
For context: I'm a mid-level data analyst/scientist who's somewhat been thrown in the deep end and asked to help build a data analytics/data science stack for my team. The company is fairly new to the game--we're just about to set up a data warehouse. I've been asked (among other things) to decide how we should structure our ETL/ELT pipelines and what tools to use given the current team's skillset (most are competent SQL users but no Python experience, though half the team are starting to learn. I would describe my Python skills to be intermediate). Ultimately I would really like our team to take ownership of ELT pipelines, not just for efficiency's sake but also to empower them with more tech skills.
I came across Meltano after reading about it in this ebook; Meltano was mentioned as an alternative to Airflow and Prefect. Meltano seems to be a good fit for us (e.g., dbt support), but from my understanding it would still require some of Airflow's scheduling and orchestration features.
12/11/2020, 3:40 PM
Hi @plain-air-55383, thanks for checking us out!
Since https://meltano.com/blog/2019/11/11/clarifying-the-target-persona-for-meltano/, we've pivoted to focus specifically on open source EL(T) (https://meltano.com/blog/2020/05/13/revisiting-the-meltano-strategy-a-return-to-our-roots/, https://meltano.com/blog/2020/05/13/why-we-are-building-an-open-source-platform-for-elt-pipelines/) with a pipelines-as-code and CLI-first approach aimed at relatively technical users as described under https://meltano.com/docs/#focus.
To illustrate, Meltano integrates with dbt and Airflow and makes it easy to set them up together, but self-hosting and deploying them in a way appropriate for your team is still left as an exercise to the reader, which can go in different directions depending on whether you already have Airflow (or something similar) set up, whether you're comfortable with Docker, what cloud you'd like to deploy to, etc. We expect users to be able to figure this out themselves right now (with guidance from other users here on Slack).
On top of that, the quality of Singer taps (extractors) for different sources varies quite a bit, and a big part of our goal with building a really great environment to develop and run them is to build a community of users (and teams) to try out and improve and maintain an ecosystem of high quality open source connectors. We're not quite there yet though, and at this stage, you're bound to run into some exceptions or other bugs with some of the less well-maintained taps, that you will need to be comfortable debugging, fixing, and upstreaming yourself (with guidance from others here in Slack) in order to get your pipelines to work reliably (and move the ecosystem forward for all of us).
@plain-air-55383 Only you can estimate where your team and needs fit inside that picture, but at this stage I generally wouldn't recommend Meltano to less technical teams.