Hi! I have a general question: what is the exact purpose of the

meltano lock

command? The docs say that it "locks plugin definition". I assume that this is different from pinning the version of the plugin. What does "plugin definition" exactly mean in this context? Thanks!

Does this link help? https://docs.meltano.com/concepts/plugins/#lock-artifacts It basically stores the plugin definition locally so the state of your project won’t change if the Hub API changes the definiton.

Thanks but I don't know what plugin definition means here. Isn't the plugin definition completely determined by the version of the plugin?

On the MeltanoHub we have additional metadata for a given plugin that is not tied to a specific version of the plugin. Many plugins may not be appropriately versioned either. https://github.com/meltano/hub/blob/main/_data/meltano/extractors/tap-github/meltanolabs.yml is an example of a plugin definition.

ok so you're saying that, even for plugins that are appropriately versioned, pinning the version may not be enough to get a completely reproducible behavior

pinning the version will guarantee the code is the same, but the metadata may change as it’s not bundled with the code. This could cause Meltano to not know how to provide config to the plugin. Locking the plugin config freezes the metadata in place so it’s fully reproducible. @Edgar Ramírez (Arch.dev) can you double check everything I’m saying here?

thanks. That's helpful. If not from the code, where does the metadata for a plugin come from?

Connectors built with our SDK are self-describing when it comes to runtime config, but they aren’t “Meltano-aware”. The metadata I linked can be thought of as Meltano-specific metadata. We curate that as part of onboarding to the Hub and have automated checks on a regular basis to keep the configs in sync as the plugin is versioned.

+1 to everything Taylor said above. Plugin metadata on the Hub is a layer on top of the packages themselves that allows Meltano, for example, to tell if a setting is sensitive, or whether it supports certain features like incremental replication. SDK-based plugins can generate metadata that's easy to translate automatically to the Hub spec, but the rest of the ecosystem relies on accurate metadata that's mostly inspected manually. The metadata can change at any given time, for SDK-based plugins this can happen with a code change in the source and for others it can happen because someone noticed drift between the package and the metadata. This is the reason why lock files exist: to provide metadata stability for the plugin. We do have an issue in the backlog to lock package versions, but I think it's unclear if we want that to happen in

meltano.yml

or in the lock file (imo the former makes more sense).

OK thanks I think I understand now. Yes indeed, this stuff is very confusing for a meltano newbie.

Sorry to dig up an older thread but I just posted https://github.com/meltano/meltano/issues/6416#issuecomment-2252792714 and wanted to follow up on what you meant by

I think it's unclear if we want that to happen in meltano.yml or in the lock file (imo the former makes more sense).

? @Edgar Ramírez (Arch.dev) Feel free to reply to the comment on the issue. 🙂

Thanks @Reuben (Matatika), I'll take a look this week.