Hi all! I am trying to setup my meltano project in a dockerfile and the container will run in a sandbox environment, so no outside APIs will be accessible (like meltanohub). I have copied both my tap and target inside my image but I am not able to install the loader when building the dockerfile. ```Loader 'target-postgres' could not be installed: failed to install plugin 'target-postgres'. #16 24.84 error: subprocess-exited-with-error #16 24.84 #16 24.84 × Preparing editable metadata (pyproject.toml) did not run successfully. #16 24.84 │ exit code: 1 #16 24.84 ╰─> [35 lines of output] #16 24.84 Traceback (most recent call last): #16 24.84 File "/redshift-to-postgres/.meltano/loaders/target-postgres/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module> #16 24.84 main() #16 24.84 File "/redshift-to-postgres/.meltano/loaders/target-postgres/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main #16 24.84 json_out['return_val'] = hook(**hook_input['kwargs']) #16 24.84 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ #16 24.84 File "/redshift-to-postgres/.meltano/loaders/target-postgres/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 181, in prepare_metadata_for_build_editable #16 24.84 return hook(metadata_directory, config_settings) #16 24.84 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ #16 24.84 File "/tmp/pip-build-env-6iv26n0x/overlay/lib/python3.11/site-packages/poetry/core/masonry/api.py", line 41, in prepare_metadata_for_build_wheel #16 24.84 poetry = Factory().create_poetry(Path(".").resolve(), with_groups=False) #16 24.84 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ #16 24.84 File "/tmp/pip-build-env-6iv26n0x/overlay/lib/python3.11/site-packages/poetry_dynamic_versioning/patch.py", line 23, in alt_poetry_create #16 24.84 name = _get_and_apply_version( #16 24.84 ^^^^^^^^^^^^^^^^^^^^^^^ #16 24.84 File "/tmp/pip-build-env-6iv26n0x/overlay/lib/python3.11/site-packages/poetry_dynamic_versioning/__init__.py", line 473, in _get_and_apply_version #16 24.84 version = _get_version(config, name) #16 24.84 ^^^^^^^^^^^^^^^^^^^^^^^^^^ #16 24.84 File "/tmp/pip-build-env-6iv26n0x/overlay/lib/python3.11/site-packages/poetry_dynamic_versioning/__init__.py", line 307, in _get_version #16 24.84 version = _get_version_from_dunamai(vcs, pattern, config) #16 24.84 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ #16 24.84 File "/tmp/pip-build-env-6iv26n0x/overlay/lib/python3.11/site-packages/poetry_dynamic_versioning/__init__.py", line 259, in _get_version_from_dunamai #16 24.84 return Version.from_vcs( #16 24.84 ^^^^^^^^^^^^^^^^^ #16 24.84 File "/tmp/pip-build-env-6iv26n0x/overlay/lib/python3.11/site-packages/dunamai/__init__.py", line 1902, in from_vcs #16 24.84 return cls._do_vcs_callback( #16 24.84 ^^^^^^^^^^^^^^^^^^^^^ #16 24.84 File "/tmp/pip-build-env-6iv26n0x/overlay/lib/python3.11/site-packages/dunamai/__init__.py", line 1936, in _do_vcs_callback #16 24.84 return callback(**kwargs) #16 24.84 ^^^^^^^^^^^^^^^^^^ #16 24.84 File "/tmp/pip-build-env-6iv26n0x/overlay/lib/python3.11/site-packages/dunamai/__init__.py", line 1058, in from_git #16 24.84 _detect_vcs(vcs) #16 24.84 File "/tmp/pip-build-env-6iv26n0x/overlay/lib/python3.11/site-packages/dunamai/__init__.py", line 354, in _detect_vcs #16 24.84 raise RuntimeError(dubious_ownership_error) #16 24.84 RuntimeError: Detected Git repository, but failed because of dubious ownership #16 24.84 [end of output] …

Hi @silverbullet1! I think it's because the target uses

poetry-dynamic-versioning

in its build environment, which relies on git to determine the package version: https://github.com/MeltanoLabs/target-postgres/blob/e756da19d07960090c8422d46cc6a021341e8ef3/pyproject.toml#L54-L56 Since you're installing from source (

-e .

),

pip

is using the build-system toml table to create build venv. Installing from a wheel doesn't have that problem

Thanks for the explanation @edgar_ramirez_mondragon. So do you mean that I should just install target-postges as a normal pip requirement instead?

If possible, yes. You should be able to pin

meltanolabs-target-postgres==0.0.5

Also if you need @silverbullet1 I can cut a release whenever, trying to wait on a couple more PRs to get merge then I"ll be doing them (tomorrow likely)

Thanks @edgar_ramirez_mondragon! I was able to get around that issue, but surprisingly getting this now 🤔

Error: Block tap-redshift not found

and the same image on my production says

[2023-07-12, 21:37:47 UTC] {pod_launcher.py:159} INFO - [2m2023-07-12T21:37:47.429128Z[0m [[32m[1mdebug    [0m] [1mInvoking: ['/redshift-to-postgres/.meltano/extractors/tap-redshift/venv/bin/tap-redshift', '--config', '/redshift-to-postgres/.meltano/run/tap-redshift/tap.f2580981-3c7d-4b91-8197-79314c8eb32b.config.json', '--discover'][0m
[2023-07-12, 21:37:47 UTC] {pod_launcher.py:159} INFO - [2m2023-07-12T21:37:47.505761Z[0m [[32m[1mdebug    [0m] [1mDeleted configuration at /redshift-to-postgres/.meltano/run/tap-redshift/tap.f2580981-3c7d-4b91-8197-79314c8eb32b.config.json[0m
[2023-07-12, 21:37:47 UTC] {pod_launcher.py:159} INFO - [2m2023-07-12T21:37:47.515975Z[0m [[31m[1merror    [0m] [1mCannot start plugin tap-redshift: Executable 'tap-redshift' could not be found. Extractor 'tap-redshift' may not have been installed yet using `meltano install extractor tap-redshift`, or the executable name may be incorrect.[0m
[2023-07-12, 21:37:47 UTC] {pod_launcher.py:159} INFO - [2m2023-07-12T21:37:47.516604Z[0m [[31m[1merror    [0m] [1mBlock run completed.          [0m [36mblock_type[0m=[35mExtractLoadBlocks[0m [36merr[0m=[35mRunnerError("Cannot start plugin tap-redshift: Executable 'tap-redshift' could not be found. Extractor 'tap-redshift' may not have been installed yet using `meltano install extractor tap-redshift`, or the executable name may be incorrect.")[0m [36mexit_codes[0m=[35m{}[0m [36mset_number[0m=[35m0[0m [36msuccess[0m=[35mFalse[0m

But I have done

meltano install extractor tap-redshift

in my dockerfile, and it ran successfully. why are both the errors different? I have a feeling that they are related 😮

No I was thinking

target-postgres

sorry I wasn't specific 😅

Ohh, sorry! Nope that doesn’t seem like an immediate blocker, we should be fine 🙂 pip install helped

You may want to force a reinstall if you changed the pip URL:

meltano install loader target-postgres --clean

so

meltano install loader target-postgres

still needs to be in dockerfile, even after pip install is there ?

Sorry, I think I got lost. Do you mean you ran

pip install

in the target venv?

I added

meltanolabs-target-postgres==0.0.5

in the

requirements.txt

and it got installed in my container. My Dockerfile and meltano.yml is still the same.

oh I see. You need to set it in

meltano.yml

, not `requirements.txt`:

plugins:
  loaders:
  - name: target-postgres
    namespace: target_postgres
    pip_url: meltanolabs-target-postgres==0.0.5

requirements.txt

is for "system-level" dependencies that should be installed alongside Meltano, i.e. any packages you want to share a virtual env with Meltano. The

pip_url

prop is for the individual actual plugin environments.

(sorry about the confusion)

Thank you so much Edgar! This was really helpful 🙂

I'll cut a release for you

That is the sweetest thing someone would do for me ❤️

just fixing a conflict

Thank you so much @visch!!