Hey all, I’ve made a little ipynb notebook utility...
# plugins-generald
daniel_walker
01/10/2022, 12:45 PMHey all, I’ve made a little ipynb notebook utility that executes notebooks and converts them to pdf, you can check it out here: https://github.com/Matatika/notebook. I’ve been using it to create reports to email out, with Apprise https://github.com/caronc/apprise doing the messaging.
Think this could be useful to others so I also wanted to check that submitting a new utility would be a MR to Meltano adding it to the discovery.yml? Couldn't immediately see a way through the hub.
a
aaronsteers
01/10/2022, 6:41 PM@daniel_walker - Very cool! Is this something you might be interested to demo for us in an upcoming #C01S996HGFK? I took a look at the code and I can see you wrapped nbconvert, which is a library I have a tiny bit of experience with - and which immediately came to mind when you mentioned this. I could see this potentially getting into
discovery.ymlaaronsteers
01/10/2022, 6:41 PMRelated: Notebooks, Visual Studio Code style
aaronsteers
01/10/2022, 6:42 PMCc @amanda.folson, @taylor
aaronsteers
01/10/2022, 6:44 PMApprise could also be super interesting as a utility plugin. Elsewhere we discussed it as a possible Singer target, but I can see value in potentially making it a meltano utility.
a
aaron_phethean
01/10/2022, 8:06 PMHi @aaronsteers Yes to both. The plan was for @daniel_walker to develop the notebook plugin and the Apprise plugins. We love the idea of scheduling them through Meltano, we've been scheduling nbconvert outside of Meltano for awhile and have a few new idea about how they could manage the notebook dependencies once configured as a plugin. Glad you like the idea!
aaron_phethean
01/10/2022, 8:08 PMOne small note on Apprise. From @daniel_walker investigations the Sendgrid option doesn't handle attachments (as it's a thin shim over the a couple of http API calls), so there's a bit of jiggling with the mail target that does get attachments working. We've wondered whether it might just be betting to have Sendgrid utility plugins that use the Sendgrid python library directly.
aaron_phethean
01/10/2022, 8:17 PM@aaronsteers two more things.
1. Where should @daniel_walker create the issue / merge requests to create a utility?
2. We were thinking about an enhancement for utility plugins to accept settings. As far as we can tell they don't right now. Same question, where should we create that issue / merge request?
a
aaronsteers
01/10/2022, 8:21 PM@aaron_phethean - Since the updates for both 1 and 2 would need to live in the meltano core libraries and/or discovery.yml, you can open them both in the main Meltano repo. New Issue · Meltano / Meltano · GitLab
Adding the the hub would probably be a later discussion, just because we haven't (yet!) been as actively promoting utilities there on the hub site.
a
aaron_phethean
01/10/2022, 8:30 PM👍
d
daniel_walker
01/11/2022, 9:15 AM@aaronsteers Thanks for the MR info 🙂. And yeah I'd love to demo this at demo day, end of the month right?
a
amanda.folson
01/11/2022, 3:01 PMYep! January 27th