trying to understand what behaviour i should expect from plugin lock files. i’m using meltano v 2.19.0 i’ve updated the

pip_urls

in my

meltano.yml

to point to a specific version of the package

extractors:
   - name: tap-postgres
     variant: transferwise
-    pip_url: git+<https://github.com/transferwise/pipelinewise-tap-postgres@297a2098b6d0818862fdaed0b5462f4d0628799b>
+    pip_url: git+<https://github.com/transferwise/pipelinewise-tap-postgres@v2.1.0>
     config:
       filter_schemas: public
       default_replication_method: INCREMENTAL
@@ -72,7 +72,7 @@ plugins:
   loaders:
   - name: target-bigquery
     variant: z3z1ma
-    pip_url: git+<https://github.com/z3z1ma/target-bigquery.git>
+    pip_url: git+<https://github.com/z3z1ma/target-bigquery.git@0.6.3>
     config:

i would have expected

meltano lock --all --update

to update the lock files for these two plugins, but it does nothing. i also tried running

meltano install --clean

but that doesn’t have any impact on the lock files either. running

meltano install

does seem to be installing the right versions, when i inspect e.g.

.meltano/extractors/tap-postgres/venv/…/site-packages

but i would also expect the lock files to be updated to reflect these new `pip_url`s. perhaps i am misunderstanding the role of the lock files based on expectations from other ecosystems like ruby or rust.

@luke_rodgers the meltano lock files are essentially locking the metadata that you retrieved from meltano hub at the time of installation. This feature was added to insulate users from the constantly updating hub metadata, since the hub is intending to represent the current metadata for the tap. Pinning the pip url in your meltano.yml is an override of whats provided by default from your lock file. The recommendation is to install a plugin, get it working, then pin the pip url in your meltano.yml. When you want to bump the pip url you should also update your lock file to retrieve updated metadata. Does that make sense?

thanks @pat_nadolny that sort of makes sense, except the part about “you should also update your lock file” is still not clear. how should i update that? after updating my

meltano.yml

i tried all the following, and none of them have any effect on the lock file(s) •

meltano lock --all --update

•

meltano install

• manually removing the existing lock file and then running

meltano add <extractor/loader> <plugin-name>

i can’t imagine i’m supposed to manually edit the lock file(s), but it is not clear to me how to get it to update otherwise

@luke_rodgers it sounds like you have the latest metadata in your lock file. What are you expecting to get updated? It sounds like youre overriding the pip url in your meltano.yml, that wont affect the lock file metadata at all but will install your pinned version

ok thanks for clarifying - i was expecting at very least the

pip_url

in the lock file to get updated, but that is probably just baggage from how lock files behave in other ecosystems.

thats fair, its a little bit confusing for sure. The lock file in meltano's context is just a snapshot of the metadata thats on the hub right now. When you run the plugin meltano will look at the lock file first then ask the hub for metadata if it doesnt exist. It was a feature that was created because some people pin to a pip url for a long time and the hub metadata could update overtime and lead to changing the behavior of their plugin without their approval. Lock files allow you to essentially snapshot the hub metadata today when you know your plugin is working as expected so its stable.