Hi All hope everyone had a good weekend I ve recently come a Meltano #troubleshooting

Hi All — hope everyone had a good weekend. I’ve re...

drew_ipson

11/22/2021, 8:37 PM

Hi All — hope everyone had a good weekend. I’ve recently come across this error in my tap-postgres’ extractors:

Copy code

import simplejson as json
ModuleNotFoundError: No module named 'simplejson'

It is a simple error fix by activating the tap’s virtual environment and running

pip install simplejson

. However, does anyone know why the setup wheels wouldn’t be installing that library during

meltano install

maarten_van_gijssel

11/23/2021, 8:13 AM

Had the same issue yesterday! It started happening when I added the following to the

meltano.yml

file:

Copy code

- name: tap-csv
      variant: meltanolabs
      pip_url: -r requirements/tap-csv-requirements.txt
      config:
        files:
          - entity: test
            path: extract/test.csv
            keys: [id]

maarten_van_gijssel

11/23/2021, 8:14 AM

I know it doesn’t make any sense, but if I remove the tap-csv extractor the tap-postgres extractor works as expected. If I add the tap-csv then suddenly the tap-postgres library no longer installs simplejson and breaks during discovery 😅

maarten_van_gijssel

11/23/2021, 8:18 AM

The error I’m getting:

Copy code

tap-postgres           | discovery | Traceback (most recent call last):
tap-postgres           | discovery |   File "/project/.meltano/extractors/postgres--core/venv/bin/tap-postgres", line 5, in <module>
tap-postgres           | discovery |     from tap_postgres import main
tap-postgres           | discovery |   File "/project/.meltano/extractors/postgres--core/venv/lib/python3.9/site-packages/tap_postgres/__init__.py", line 14, in <module>
tap-postgres           | discovery |     import tap_postgres.sync_strategies.common as sync_common
tap-postgres           | discovery |   File "/project/.meltano/extractors/postgres--core/venv/lib/python3.9/site-packages/tap_postgres/sync_strategies/common.py", line 2, in <module>
tap-postgres           | discovery |     import simplejson as json
tap-postgres           | discovery | ModuleNotFoundError: No module named 'simplejson'
tap-postgres           | discovery |
meltano                | elt       | DEBUG Deleted configuration at /project/.meltano/run/elt/postgres--core-to-snowflake/e4f0c283-156f-4d2e-b620-b489bc4cfc60/target.ae94275b-ccd8-49e9-9cce-854f4f75f2c6.config.json
meltano                | elt       | DEBUG Deleted configuration at /project/.meltano/run/elt/postgres--core-to-snowflake/e4f0c283-156f-4d2e-b620-b489bc4cfc60/tap.f95a6c4a-aaf3-495b-bd1b-456e9779e893.config.json
meltano                | elt       | DEBUG ELT could not be completed: Cannot start extractor: Catalog discovery failed: command ['/project/.meltano/extractors/postgres--core/venv/bin/tap-postgres', '--config', '/project/.meltano/run/elt/postgres--core-to-snowflake/e4f0c283-156f-4d2e-b620-b489bc4cfc60/tap.f95a6c4a-aaf3-495b-bd1b-456e9779e893.config.json', '--discover'] returned 1

maarten_van_gijssel

11/23/2021, 8:24 AM

If I install simplejson into the venv of the extractor and run a pipeline I get another error:

Copy code

postgres--core   | extractor | time=2021-11-23 08:23:51 name=tap_postgres level=CRITICAL message=Type is not JSON serializable: decimal.Decimal
postgres--core   | extractor | Traceback (most recent call last):
postgres--core   | extractor |   File "/project/.meltano/extractors/postgres--core/venv/bin/tap-postgres", line 8, in <module>
postgres--core   | extractor |     sys.exit(main())
postgres--core   | extractor |   File "/project/.meltano/extractors/postgres--core/venv/lib/python3.9/site-packages/tap_postgres/__init__.py", line 457, in main
postgres--core   | extractor |     raise exc
postgres--core   | extractor |   File "/project/.meltano/extractors/postgres--core/venv/lib/python3.9/site-packages/tap_postgres/__init__.py", line 454, in main
postgres--core   | extractor |     main_impl()
postgres--core   | extractor |   File "/project/.meltano/extractors/postgres--core/venv/lib/python3.9/site-packages/tap_postgres/__init__.py", line 443, in main_impl
postgres--core   | extractor |     do_sync(conn_config, args.catalog.to_dict() if args.catalog else args.properties,
postgres--core   | extractor |   File "/project/.meltano/extractors/postgres--core/venv/lib/python3.9/site-packages/tap_postgres/__init__.py", line 321, in do_sync
postgres--core   | extractor |     state = sync_traditional_stream(conn_config,
postgres--core   | extractor |   File "/project/.meltano/extractors/postgres--core/venv/lib/python3.9/site-packages/tap_postgres/__init__.py", line 181, in sync_traditional_stream
postgres--core   | extractor |     state = full_table.sync_table(conn_config, stream, state, desired_columns, md_map)
postgres--core   | extractor |   File "/project/.meltano/extractors/postgres--core/venv/lib/python3.9/site-packages/tap_postgres/sync_strategies/full_table.py", line 156, in sync_table
postgres--core   | extractor |     singer.write_message(record_message)
postgres--core   | extractor |   File "/project/.meltano/extractors/postgres--core/venv/lib/python3.9/site-packages/singer/messages.py", line 300, in write_message
postgres--core   | extractor |     sys.stdout.buffer.write(format_message(message))
postgres--core   | extractor |   File "/project/.meltano/extractors/postgres--core/venv/lib/python3.9/site-packages/singer/messages.py", line 296, in format_message
postgres--core   | extractor |     return orjson.dumps(message.asdict(), option=orjson.OPT_APPEND_NEWLINE)
postgres--core   | extractor | TypeError: Type is not JSON serializable: decimal.Decimal

maarten_van_gijssel

11/23/2021, 8:25 AM

If I Google for ” TypeError: Type is not JSON serializable: decimal.Decimal” the recommendation is to use simplejson 🤣

maarten_van_gijssel

11/23/2021, 8:26 AM

So no clue what’s exactly going on here, but somehow the installation of an extractor is influencing the installation of another extractor

maarten_van_gijssel

11/23/2021, 8:52 AM

Debugging this further it looks like a fresh install of Meltano does not install the required simplejson dependency for tap-postgres

maarten_van_gijssel

11/23/2021, 8:52 AM

Due to the sudden change of this, my guess is that this is due to a newer version of pip

maarten_van_gijssel

11/23/2021, 8:52 AM

When Meltano installs it will always use the latest version of pip which might result in different installation behaviour

maarten_van_gijssel

11/23/2021, 9:02 AM

Looked into a previously built docker image which works, but seems that that particular docker image has the same latest pip version (21.3.1) and the one that’s currently failing.

maarten_van_gijssel

11/23/2021, 9:02 AM

There’s a difference in Meltano version 1.88.0 and 1.87.1, so checking if that might be the issue

maarten_van_gijssel

11/23/2021, 9:04 AM

Also not an issue 😅

maarten_van_gijssel

11/23/2021, 9:12 AM

Think I’m just going to diff the installed packages between the working and not working installations 🤔

maarten_van_gijssel

11/23/2021, 9:23 AM

The diff

maarten_van_gijssel

11/23/2021, 9:24 AM

My guess is going to be the

pipelinewise_singer_python-1.4.0.dist-info

maarten_van_gijssel

11/23/2021, 9:25 AM

See what happens if I downgrade that

maarten_van_gijssel

11/23/2021, 9:27 AM

Diff of the

pip list

maarten_van_gijssel

11/23/2021, 9:29 AM

Installing 1.3.0 of

pipelinewise-singer-python

actually installs simplejson!

maarten_van_gijssel

11/23/2021, 9:40 AM

Verified this is the fix! If you change your

pip_url

in your meltano.yml to include

pipelinewise-singer-python==1.3.0

the extractor works as expected.

maarten_van_gijssel

11/23/2021, 9:41 AM

I’ve created a PR to have this fixed upstream https://github.com/transferwise/pipelinewise-tap-postgres/pull/134

peter_kosztolanyi

11/23/2021, 10:58 AM

hey, thanks for reporting it. This is caused by an incorrect version number given to a new version of the upstream

pipelinewise-singer-python

. We deleted the incorrect one from PyPI and will re-release it shortly as 2.0.0. As you highlighted it turns out it’s not fully backward compatible. Reinstalling the tap should fix the problem, let me know if it’s not the case. We will roll out 2.0.0 to the supported taps and targets separately one-by-one.

boggdan_barrientos

11/23/2021, 2:31 PM

tap-mysql

was affected too, I think all

transferwise

variant, but reinstalling fixed it.

drew_ipson

11/23/2021, 5:02 PM

@maarten_van_gijssel Thank you for your awesome work on this!

Open in Slack

Previous Next