Hi < micael bergeron> I believe that you contributed to melt Meltano #announcements

Hi <@U06D0S3FRTJ> , I believe that you contributed...

tuan_cao

12/16/2019, 6:16 AM

Hi @micael_bergeron , I believe that you contributed to meltano target-postgres. Could you help me out with this issue: I keep having this error:

Copy code

Traceback (most recent call last):
  File "target-psql/bin/target-postgres", line 11, in <module>
    load_entry_point('target-postgres==1.1.3', 'console_scripts', 'target-postgres')()
  File "/Users/tuancao/.virtualenvs/target-psql/lib/python3.7/site-packages/target_postgres-1.1.3-py3.7.egg/target_postgres/__init__.py", line 138, in main
  File "/Users/tuancao/.virtualenvs/target-psql/lib/python3.7/site-packages/target_postgres-1.1.3-py3.7.egg/target_postgres/__init__.py", line 100, in persist_lines
  File "/Users/tuancao/.virtualenvs/target-psql/lib/python3.7/site-packages/target_postgres-1.1.3-py3.7.egg/target_postgres/db_sync.py", line 116, in __init__
KeyError: 'schema'
....

tuan_cao

12/16/2019, 6:18 AM

What I did is piping tap-gitlab (the one from meltano) to target-postgres (which I down load from github and run

python setup.py install

. Using following config.json file as below:

Copy code

{
  "postgres_host": "localhost",
  "postgres_port": 54320,
  "postgres_database": "postgres",
  "postgres_username": "postgres",
  "postgres_password": "123456",
  "postgres_schema": "gitlab",
  "invalid_records_threshold": 10000
}

tuan_cao

12/16/2019, 6:21 AM

below is the list of command line i used:

Copy code

mkvirtualenv target-psql
cd target-psql
git clone <https://github.com/meltano/target-postgres.git>
cd target-postgres
python setup.py install
cd ..
deactivate
tap-gitlab/bin/tap-gitlab -c gitlab-config.json | target-psql/bin/target-postgres -c postgres_config.json >> state.json 
workon target-psql
cdvirtualenv
pip install psycopg2-binary
tap-gitlab/bin/tap-gitlab -c gitlab-config.json | target-psql/bin/target-postgres -c postgres_config.json >> state.json

tuan_cao

12/16/2019, 6:22 AM

what puzzle me is that, if I use the same technique but using official singer-target-postgres, the mentioned error does not occur, and I saw some data can be inserted to my database (Local Postgres)

tuan_cao

12/16/2019, 6:24 AM

Copy code

mkvirtualenv singer-target-psql
cdvirtualenv
pip install singer-target-postgres
cd ..
deactivate
tap-gitlab/bin/tap-gitlab -c gitlab-config.json | singer-target-psql/bin/target-postgres -c postgres_config.json >> state.json

tuan_cao

12/16/2019, 8:01 AM

Actually I figured out why, it's because config.json file that meltano's target_postgres use different

key

from official one. below config correct the issue:

Copy code

{
  "host": "localhost",
  "port": 54320,
  "dbname": "postgres",
  "user": "postgres",
  "password": "123456",
  "schema": "gitlab",
  "invalid_records_threshold": 10000
}

micael_bergeron

12/18/2019, 4:34 PM

he @tuan_cao sorry for the late reply, glad you found out! Basically the official

target-postgres

changed to use the datamill implementation IIRC. We are still running off a fork of the old one, which explains why the configuration is not the same.

tuan_cao

12/18/2019, 4:54 PM

ah, I see! Good to know. Thanks @micael_bergeron!

12 Views

Open in Slack

Previous Next