<@U06BX9QQ4A3> <@U06CBKL5W7L> just checking the wa...
# meltano-plugin-development
f
@edgar_ramirez_mondragon @aaronsteers just checking the water. How we feeling about users importing parts of meltano ? Last time we chatted about this was in the context of a v2 api.
I don't have any huge need for this yet, but theres some potentially useful stuff in meltano.core. Right now mostly thinking of things like shared logging code, potentially stdout/stderr handling stuff - that could be handy to share with an extension SDK.
Those are all things we could spin out into a general purpose meltano utility package if we really needed to DRY things up but didn't want folks importing from meltano.core.
e
I think that’s the desirable approach. Either have them import from something like
meltano.extend
with public APIs or some common library both Meltano and plugins import, but I feel like the latter just means more work for us.
w
I think most users would expect it's 100% okay to use any attribute under
meltano
that is included in a module's
__all__
, does not begin with an underscore, and is not part of
meltano.core
.
That's not exactly how we've designed the Meltano Python package, but with Python and its inability to mark things as private or exposed, what our users assume reflects how they'll use it.
f
i can drop some more thoughts after lunch but during the last convo the discussion wasn't "can they" ..which yea they obviously can , its "should they", because meltano.core makes no claims about stability and a lot of our "internal" api's have under gone refactors that are not reflected in our versioning.
a
Those are all things we could spin out into a general purpose meltano utility package if we really needed to DRY things up but didn't want folks importing from meltano.core.
I think that’s the desirable approach. ... have them import from something like
meltano.extend
with public APIs...
This is the direction I'd lean towards also. Ideally the
meltano
namespace would be clean but it's not as of now. Also, even if it were clean, I'd feel better about dependencies on a helper library or helper module for this specific use case - rather than depending directly on the meltano python libraries as an external API (read: an API needing to remain stable).
There's another question about bundling and the implications of plugins importing all of the meltano libraries and dependencies. A
meltano-core
or
meltano-utils
or something like that with minimal external dependencies would be ideal when it comes to what another Python library might reasonably import for interop reasons.
Also - I guess that's another thing to tease out here: are we doing this for interop/convenience of matching specs, or for invocation. There are good reasons, I think, for plugins (and the SDK) to not have too many special hooks and instead prefer the public CLI interface when it needs to directly invoke Meltano.
f
"interop/convenience of matching specs, or for invocation." my immediate thought was more along the lines of "how much can I copy and paste from meltano before someone tells me this isn't pythonic"
mostly because in Go land:

https://www.youtube.com/watch?v=PAAkCSZUG1c&amp;t=568s

"a little copying is better than a little dependency"
a
Thanks for sharing this. I'm a little wiser today than yesterday. 🧠
f
side note, i was at that talk - there was a group of Java folks sitting next to us that basically grumbled non stop. it was super entertaining.
a
hahaha
GoLang folks troll the rest of us by prioritizing stable, predictable, and atomic over DRY and spider-webby