https://linen.dev logo
Join Slack
Powered by
Hello! I’m wondering if there is a way to know whe...
# random
f
Hello! I’m wondering if there is a way to know when a job ends inside a scheduler. I have set a scheduled job to run once a day and i want to trigger a message to api/email or somewhere if it failed or finish successfully (I’m guessing to add a custom script at the end of the job but if it fails for some reason it won’t be executed).
e
Most schedulers should have a way of triggering something like that, either built-in or by calling some third-party. For example, I use GitHub Actions and I added steps at the beginning and end of a Meltano pipeline to ping healthchecks.io, which notifies me when the daily scheduled run doesn't execute to completion: https://github.com/edgarrmondragon/meltano-dogfood/blob/main/.github%2Fworkflows%2Fmelty.yaml#L259-L266 Or do you mean something different?
f
Hmmm you’re using the Github Actions to schedule the jobs, i was following the documentation and trying to do it using the orchestrator airflow. Do you know if there a way to achieve the same of you using it or maybe execute a custom script that make the call ? Which approach do you recommend? I’m quite new here😅, thanks in advance!
@Edgar Ramírez (Arch.dev) Another question, if for some reason in the flow something fails, is there a way to catch the error and forward it through a slack message or make an api call to notify?
e
Hmmm you’re using the Github Actions to schedule the jobs, i was following the documentation and trying to do it using the orchestrator airflow.
Yeah, I use GHA in my dogfood project and it's enough for my needs but for teams or orgs that need to orchestrate a lot of things in a more complex manner, something like Airflow is probably better.
Do you know if there a way to achieve the same of you using it or maybe execute a custom script that make the call ? Which approach do you recommend?
If you'd still like using Airflow, it's be fairly well supported pattern: https://airflow.apache.org/docs/apache-airflow/stable/howto/notifications.html
if for some reason in the flow something fails, is there a way to catch the error and forward it through a slack message or make an api call to notify?
Not at the moment. Another member of the community opened https://github.com/meltano/meltano/issues/8708, but we currently don't have a clear path to get there. But do give the issue 👍 if you'd like what it describes, or comment with your own ideas and thoughts!
✅ 1
👀 1
a
Hi @Facundo Miño - if you are after easy to use meltano orchestration with notifications. Give Matatika-CE or our cloud a go? https://github.com/Matatika/matatika-ce Your existing meltano project jobs will just run when you create a workspace from your git repository. You can watch these pipelines for errors or get alerts for everything. Cheers, Aaron
s
➕ On implementing this feature. Would love to find meltano's main.py and simply plan an init of sentry in the middle 😆
e
@Stéphane Burwash have you seen meltano#8708?
s
Thanks for the headsup @Edgar Ramírez (Arch.dev)! I'll look into it 😄
Copy code
source: |
      event, err = parse_regex(.line.event, r'METRIC: (?P<json>\{.*\})', numeric_groups: false)
      if (event !=null && exists(event.json)) {
          json_str = to_string(event.json)
          parsed_json, err = parse_json(json_str)
          if (parsed_json != null) {
              .line.metric = parsed_json
          }
      }
TADAAAAAAH, I got me some meltano json logs!
If you want to add this somewhere in your docs @Edgar Ramírez (Arch.dev) 😄
👀 1
(This is in VRL)
6 Views
Open in Slack
PreviousNext
Powered by