https://linen.dev logo
#random
Title
# random
c

clement

07/01/2023, 11:44 PM
Since I like meltano a lot, I'de like to share first user feedback. For context, i have ~15y experience in dev but very little python experience (3y on an already setup project) and I built a custom etl tool in Typescript (i know, i know) that works ok but is very difficult to extend so I was looking for a set of open tools that could replace it, I found the singer + airflow + dbt combo to be a good fit and then I found you ❣️ Atm i'm trying some key features I need to validate meltano before a rewrite and when creating a new project, I found some rough edges. So here's where I got stuck: • difficulty setting up a new project: I knew I would not use pip as I wanted something to handle venvs + dep locking + a python version independent from my system's version + dependencies in the project directory to be closer to what I know in js where all this is handled easily. So I had to learn about the various types of setup like setup.py, pipenv, pyproject.toml, etc. That was confusing. Ofc, I didn't understand any of these fully so I did go with Pipenv at first and later moved to poetry when I saw that the custom extractor template used poetry. I did loose a lot of time configuring devcontainers too. Smoothing this should be easy enough with a more battery included meltano init or a custom base template with poetry + devcontainers • Incompatibility with python 3.11.4 that i discovered on the field, not super important but a bit deceptive. Also fixed with a tested template probably • Created a new extractor tap, the creation went super smoothly but I wanted to add it to my local repo and couldn't find where to put it (don't understand the dir structure yet) and how to make meltano know about it. Did find the fix but it took some time • Now stuck on 3 things: 1) can't filter out a column from an extractor, tried select_filter but doesn't seem to work for some reason 2) I would like to change the generated type for a column in my db for perf reasons, seems a bit involved 3) would like to have my dbt source file generated from the tap catalog but couldn't find how All in all, I can clearly see the huge potential of your amazing toolchain 🔥, and I hope that sharing my journey will help make meltano a better tool. Lmk if I should keep on editing this as I go along my journey