getting below error whenever I try `meltano add l...
# troubleshooting
s
getting below error whenever I try
meltano add loader target-postgres
Copy code
Loader 'target-postgres' could not be installed: failed to install plugin 'target-postgres'.
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [23 lines of output]
      running egg_info
      creating /tmp/pip-pip-egg-info-x_69i0ar/psycopg2_binary.egg-info
      writing /tmp/pip-pip-egg-info-x_69i0ar/psycopg2_binary.egg-info/PKG-INFO
      writing dependency_links to /tmp/pip-pip-egg-info-x_69i0ar/psycopg2_binary.egg-info/dependency_links.txt
      writing top-level names to /tmp/pip-pip-egg-info-x_69i0ar/psycopg2_binary.egg-info/top_level.txt
      writing manifest file '/tmp/pip-pip-egg-info-x_69i0ar/psycopg2_binary.egg-info/SOURCES.txt'

      Error: pg_config executable not found.

      pg_config is required to build psycopg2 from source.  Please add the directory
      containing pg_config to the $PATH or specify the full executable path with the
      option:

          python setup.py build_ext --pg-config /path/to/pg_config build ...

      or with the pg_config option in 'setup.cfg'.

      If you prefer to avoid building psycopg2 from source, please install the PyPI
      'psycopg2-binary' package instead.

      For further information please check the 'doc/src/install.rst' file (also at
      <<https://www.psycopg.org/docs/install.html>>).

      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Installed 1/2 plugins
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Failed to install plugin(s)
m
are you on python 3.10? I had the same issue earlier this week
c
sudo pip install psycopg2
That sounds like a pretty bad idea
Error: pg_config executable not found.
the
pg_config
binary is part of postgres. You will need to have some form of postgres libraries installed in order for the machine where you run meltano to have successful connectivity to a Postgres server.
Now, the reason that the
target-postgres
is looking for
pg_config
in the first place is because the Python package for talking to Postgres servers comes in two different formats, a pre-compiled format and a sourcecode format. Normally, the pre-compiled format of the Python package (
psycopg2-binary
) should install just fine. However, the version of
target-postgres
that is published in Pypi.org has not been updated in a while. There have been some updates to
target-postgres
which you can take advantage of by doing the following: 1.
meltano add loader target-postgres --no-install
<- this registers the loader in your project but doesn't immediately go ahead with the installation 2. Edit the
meltano.yml
file and replace the
pip_url
for
target-postres
to point to the pull request that has the latest version of the
psycopg2-binary
dependency:
Copy code
--- meltano.yml.orig    2022-12-08 05:28:23.537027275 +0000
+++ meltano.yml 2022-12-08 05:25:34.471202873 +0000
@@ -9,4 +9,4 @@
   loaders:
   - name: target-postgres
     variant: transferwise
-    pip_url: pipelinewise-target-postgres
+    pip_url: git+<https://github.com/transferwise/pipelinewise-target-postgres/@refs/pull/106/head>
3.
meltano install
.... happy times
c
I just tested my instructions verbatim and it worked fine for me. Can you post your error messages?
s
here is my config for meltano.yml for loader
Copy code
loaders:
  - name: target-postgres
    variant: transferwise
    pip_url: pipelinewise-target-postgres
    config:
      host: <http://XXX.XXX.XXX.XXX|XXX.XXX.XXX.XXX>
      dbname: XXXXXXXX
      user: XXXXXXXX
      default_target_schema: XXXXXXXX
      #  transformers:
      #  - name: dbt
      #pip_url: dbt==0.20.0
      #files:
      #- name: dbt
      pip_url: git+<https://github.com/transferwise/pipelinewise-target-postgres/@refs/pull/106/head>
error is
Copy code
To add it to your project another time so that each can be configured differently,
add a new plugin inheriting from the existing one with its own unique name:
	meltano add loader target-postgres--new --inherit-from target-postgres


To learn more about loader 'target-postgres', visit <https://hub.meltano.com/loaders/target-postgres--transferwise>
Loader 'target-postgres' already exists in your Meltano project
root@fth-dev-gpm-postgress:/home/meltano/meltano-dev/fth-dev1# meltano add loader target-postgres
To add it to your project another time so that each can be configured differently,
add a new plugin inheriting from the existing one with its own unique name:
	meltano add loader target-postgres--new --inherit-from target-postgres

Loader 'target-postgres' already exists in your Meltano project
Installing loader 'target-postgres'...
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [23 lines of output]
      running egg_info
      creating /tmp/pip-pip-egg-info-wqn7epk7/psycopg2_binary.egg-info
      writing /tmp/pip-pip-egg-info-wqn7epk7/psycopg2_binary.egg-info/PKG-INFO
      writing dependency_links to /tmp/pip-pip-egg-info-wqn7epk7/psycopg2_binary.egg-info/dependency_links.txt
      writing top-level names to /tmp/pip-pip-egg-info-wqn7epk7/psycopg2_binary.egg-info/top_level.txt
      writing manifest file '/tmp/pip-pip-egg-info-wqn7epk7/psycopg2_binary.egg-info/SOURCES.txt'

      Error: pg_config executable not found.

      pg_config is required to build psycopg2 from source.  Please add the directory
      containing pg_config to the $PATH or specify the full executable path with the
      option:

          python setup.py build_ext --pg-config /path/to/pg_config build ...

      or with the pg_config option in 'setup.cfg'.

      If you prefer to avoid building psycopg2 from source, please install the PyPI
      'psycopg2-binary' package instead.

      For further information please check the 'doc/src/install.rst' file (also at
      <<https://www.psycopg.org/docs/install.html>>).

      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Loader 'target-postgres' could not be installed: failed to install plugin 'target-postgres'.
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Failed to install plugin(s)
c
here is my config for meltano.yml for loader
You didn't replace the
pip_url
with the URL. It still says
pip_url: pipelinewise-target-postgres
in your meltano.yml
s
Thanks, It went through , really appreciated your help @christoph
c
Awesome. Great that it worked. As soon as Meltano's own
target-postgres
becomes the default variant in the Hub, this kind of intervention should no longer be required.
m
The same issue is true of the target-redshift loader (which I suppose isn’t surprising). I was able to follow Christoph’s steps above to resolve it. Specifically, my
meltano.yml
file contains:
Copy code
- name: target-redshift
    variant: transferwise
    pip_url: git+<https://github.com/transferwise/pipelinewise-target-redshift/@refs/pull/202/head>