https://linen.dev logo
Join Slack
Powered by
What are you all using for visibility into your Me...
# best-practices
q
What are you all using for visibility into your Meltano runs, esp. now that the UI is being deprecated?
t
vim and sqlite3 😛
c
vim and sqlite3
You should try adding vim-dadbod and vim-dadbod-ui. I was a bit hesitant about any 'ui' plugins ... but this one actually does a lot of things right!
@quinn_batten: I'm on the Dagster bandwagon now. Especially since Jules released the extensions and modules to make it straight-forward to load all your meltano jobs and schedules into dagster.
q
@christoph Ooh interesting. We are experimenting w dagster for one small project to see if we want to use it more widely
I think my short term fix for now is [watch airflow webserver UI] + [trawl thru log files on failure]. It’s not very nice tho, this is really a large loss of functionality for us
c
My Dagster usage is really simple at the moment: 1. Log in and check status of scheduled jobs 2. Manually launch jobs when needed 3. Check meltano job logs
s
@quinn_batten would you mind elaborating a bit - like describe a common workflow you had within the UI? What exactly did you make visible with the UI 😄 I'm certain @taylor would love to know this as well.
q
I mostly used the UI as a one-stop place to make sure that all the pipelines are running ok, and if any failed, to take a quick peek at the logs and see what went wrong. I have ~20 pipelines and they’re pulling from several different sources so its really nice to be able to check the status of all of em quickly
I can also check airflow-webserver for job failures, but then when something fails, in order to diagnose, i have to dig around for the log files (unless i’m missing something, which i might be)
So the problem I’m facing rn, which I’d love ideas about how to solve, is: How do I easily check the status of every pipeline in my Meltano project? And if & when something goes wrong, how do I easily access the logs for that run?
c
Some examples from a Dagster test environment
a
I love the Dagster view! While this is still a little ways away, I do want to share and request feedback in the GitHub discussion regarding on our future spec for logs. We'd like to build this in a way that the CLI would give summary as well as detailed info, and the output could be wrapped in a nice point-and-click UI down the road: https://github.com/meltano/meltano/discussions/6850
s
Thank you @quinn_batten! That makes it way easier to understand. I'm wondering whether there isn't some way of exposing that "monitoring" functionality that currently exists inside the UI in a simple way on a CLI/API-level to cover this use case? Or maybe it already is? I'm thinking about like a bash script or something like that? @aaronsteers ?
q
Getting a nice summary of pipelines from the CLI would be nifty, and would cover my needs from the UI!
a
I'm wondering whether there isn't some way of exposing that "monitoring" functionality that currently exists inside the UI in a simple way
None that I'm aware of. But definitely as we build out the CLI we could reuse some of that code.
s
@aaronsteers hm ok, but there is some functionality inside the UI right? It's hard to imagine that this involves so much frontend code, don't you think we could just have a bash script that does the same thing as the frontend does? Or is it more involved than I imagine?
a
Yeah - it's tied in frontend code but also native python calls. Someone could certainly go in and try to reverse engineer or extract the logic - but it wouldn't necessarily be faster than just building the CLI feature 🙂
s
Ok I get it 😄
Open in Slack
PreviousNext
Powered by