Hi All , I am trying to download and maintain my own copy of taps and targets to avoid any changes affecting the version we will be using, our git doesn't allow pip_url approach as it is disabled , I am trying to use the local path --from-ref approach but it needs plugin definition and I see most of the taps and targets dont have this definition in them for example https://github.com/MeltanoLabs/tap-slack https://github.com/MeltanoLabs/target-snowflake both of these reports are missing plugin definitions when I try the yml i have in Snowflake it says invalid definition missing name and namespace. Can anyone suggest to me how to handle this? My target is • Clone tap to extract the folder as tap-ABCDE • Add this Tap as meltano add extractor --from-ref path/to/cloned/repo/tap-ABCD-def.yml • keep adding and maintain a local repo

I don't really understand your requirement. If you just want to run with plugins you have cloned locally, you can first add them via

meltano add extractor tap-slack --no-install

or as a custom plugin instead (no

.lock

file)

meltano add extractor --from-ref <https://raw.githubusercontent.com/meltano/hub/main/_data/meltano/extractors/tap-slack/meltanolabs.yml> tap-slack --no-install

and then change its

pip_url

in the

meltano.yml

to

path/to/cloned/repo/tap-slack

, followed by running

meltano install --clean extractor tap-slack

Awesome, this is what I need, let me try this. Thanks for your time.

Also by "not allow" the pip url approach that's not a thing unless you literally can't clone the repo yourself

pip_url from my company git repo is not allowed, yes we can still use them with SSH but our security team has asked us to stop cloning from URL(company repo) .not sure what is the underlying situation there .

@Siddu Hussain ahh it's a security control. In that case I'd recommend not working around the control and instead publishing each tap/target you have to a registry that you all have internally (internal pip registry) 🤷 glad you have a solutino

I’m a fan of @visch idea. Add a private meltano hub and you have an awesome secure enterprise etl registry solution. https://medium.com/@rfrankel_8960/a-private-meltano-hub-for-enterprise-users-b23c324632d0

Good idea, but it doesn't solve the security concern they have which is they want to be sure they control all the code they run so they require everyone to vendor their own dependencies 🤷

I was wondering why we don't have pypi publish for the taps and targets, at least for once maintained by meltano

Right. Not entirely out of the box. You’d have to publish your own plugins with url to your private pypi. It’s a place everyone in the org can go to for the secure and approved extractors and loaders.

@Reuben (Matatika) - How can I generate the plugin definition for custom plugins. I don't see it documented. Something like this.

<https://raw.githubusercontent.com/meltano/hub/main/_data/meltano/extractors/tap-slack/meltanolabs.yml>

There's no way to generate it currently, you will have to piece that together yourself - see https://github.com/meltano/sdk/issues/253 https://github.com/meltano/sdk/issues/135 If you are working with a local

tap-slack

(or another plugin where the definition is already available), for the most part you can just copy that file and make the changes you need.