sameel_ahamed
12/07/2022, 9:43 PMmeltano add loader target-postgres
Loader 'target-postgres' could not be installed: failed to install plugin 'target-postgres'.
error: subprocess-exited-with-error
× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [23 lines of output]
running egg_info
creating /tmp/pip-pip-egg-info-x_69i0ar/psycopg2_binary.egg-info
writing /tmp/pip-pip-egg-info-x_69i0ar/psycopg2_binary.egg-info/PKG-INFO
writing dependency_links to /tmp/pip-pip-egg-info-x_69i0ar/psycopg2_binary.egg-info/dependency_links.txt
writing top-level names to /tmp/pip-pip-egg-info-x_69i0ar/psycopg2_binary.egg-info/top_level.txt
writing manifest file '/tmp/pip-pip-egg-info-x_69i0ar/psycopg2_binary.egg-info/SOURCES.txt'
Error: pg_config executable not found.
pg_config is required to build psycopg2 from source. Please add the directory
containing pg_config to the $PATH or specify the full executable path with the
option:
python setup.py build_ext --pg-config /path/to/pg_config build ...
or with the pg_config option in 'setup.cfg'.
If you prefer to avoid building psycopg2 from source, please install the PyPI
'psycopg2-binary' package instead.
For further information please check the 'doc/src/install.rst' file (also at
<<https://www.psycopg.org/docs/install.html>>).
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Installed 1/2 plugins
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.
Failed to install plugin(s)
Matt Menzenski
12/07/2022, 9:52 PMMatt Menzenski
12/07/2022, 10:00 PMchristoph
12/08/2022, 5:09 AMsudo pip install psycopg2That sounds like a pretty bad idea
christoph
12/08/2022, 5:21 AMError: pg_config executable not found.the
pg_config
binary is part of postgres. You will need to have some form of postgres libraries installed in order for the machine where you run meltano to have successful connectivity to a Postgres server.christoph
12/08/2022, 5:28 AMtarget-postgres
is looking for pg_config
in the first place is because the Python package for talking to Postgres servers comes in two different formats, a pre-compiled format and a sourcecode format.
Normally, the pre-compiled format of the Python package (psycopg2-binary
) should install just fine.
However, the version of target-postgres
that is published in Pypi.org has not been updated in a while.
There have been some updates to target-postgres
which you can take advantage of by doing the following:
1. meltano add loader target-postgres --no-install
<- this registers the loader in your project but doesn't immediately go ahead with the installation
2. Edit the meltano.yml
file and replace the pip_url
for target-postres
to point to the pull request that has the latest version of the psycopg2-binary
dependency:
--- meltano.yml.orig 2022-12-08 05:28:23.537027275 +0000
+++ meltano.yml 2022-12-08 05:25:34.471202873 +0000
@@ -9,4 +9,4 @@
loaders:
- name: target-postgres
variant: transferwise
- pip_url: pipelinewise-target-postgres
+ pip_url: git+<https://github.com/transferwise/pipelinewise-target-postgres/@refs/pull/106/head>
3. meltano install
.... happy timessameel_ahamed
12/08/2022, 5:33 AMchristoph
12/08/2022, 5:35 AMsameel_ahamed
12/08/2022, 5:37 AMloaders:
- name: target-postgres
variant: transferwise
pip_url: pipelinewise-target-postgres
config:
host: <http://XXX.XXX.XXX.XXX|XXX.XXX.XXX.XXX>
dbname: XXXXXXXX
user: XXXXXXXX
default_target_schema: XXXXXXXX
# transformers:
# - name: dbt
#pip_url: dbt==0.20.0
#files:
#- name: dbt
pip_url: git+<https://github.com/transferwise/pipelinewise-target-postgres/@refs/pull/106/head>
sameel_ahamed
12/08/2022, 5:38 AMTo add it to your project another time so that each can be configured differently,
add a new plugin inheriting from the existing one with its own unique name:
meltano add loader target-postgres--new --inherit-from target-postgres
To learn more about loader 'target-postgres', visit <https://hub.meltano.com/loaders/target-postgres--transferwise>
Loader 'target-postgres' already exists in your Meltano project
root@fth-dev-gpm-postgress:/home/meltano/meltano-dev/fth-dev1# meltano add loader target-postgres
To add it to your project another time so that each can be configured differently,
add a new plugin inheriting from the existing one with its own unique name:
meltano add loader target-postgres--new --inherit-from target-postgres
Loader 'target-postgres' already exists in your Meltano project
Installing loader 'target-postgres'...
error: subprocess-exited-with-error
× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [23 lines of output]
running egg_info
creating /tmp/pip-pip-egg-info-wqn7epk7/psycopg2_binary.egg-info
writing /tmp/pip-pip-egg-info-wqn7epk7/psycopg2_binary.egg-info/PKG-INFO
writing dependency_links to /tmp/pip-pip-egg-info-wqn7epk7/psycopg2_binary.egg-info/dependency_links.txt
writing top-level names to /tmp/pip-pip-egg-info-wqn7epk7/psycopg2_binary.egg-info/top_level.txt
writing manifest file '/tmp/pip-pip-egg-info-wqn7epk7/psycopg2_binary.egg-info/SOURCES.txt'
Error: pg_config executable not found.
pg_config is required to build psycopg2 from source. Please add the directory
containing pg_config to the $PATH or specify the full executable path with the
option:
python setup.py build_ext --pg-config /path/to/pg_config build ...
or with the pg_config option in 'setup.cfg'.
If you prefer to avoid building psycopg2 from source, please install the PyPI
'psycopg2-binary' package instead.
For further information please check the 'doc/src/install.rst' file (also at
<<https://www.psycopg.org/docs/install.html>>).
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Loader 'target-postgres' could not be installed: failed to install plugin 'target-postgres'.
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.
Failed to install plugin(s)
christoph
12/08/2022, 5:40 AMhere is my config for meltano.yml for loaderYou didn't replace the
pip_url
with the URL. It still says pip_url: pipelinewise-target-postgres
in your meltano.ymlsameel_ahamed
12/08/2022, 5:59 AMchristoph
12/08/2022, 6:56 AMtarget-postgres
becomes the default variant in the Hub, this kind of intervention should no longer be required.Matt Menzenski
12/22/2022, 5:19 AMmeltano.yml
file contains:
- name: target-redshift
variant: transferwise
pip_url: git+<https://github.com/transferwise/pipelinewise-target-redshift/@refs/pull/202/head>