After bumping meltano to 2 6 0 all of our tasks are failing Meltano #troubleshooting

After bumping meltano to 2.6.0, all of our tasks a...

martin_morset

09/14/2022, 6:43 AM

After bumping meltano to 2.6.0, all of our tasks are failing because the tasks are trying to call out to hub.meltano.com. It didn't need to before, and the firewall will prevent this from happening. I wish there was a switch to disable all outside communication from meltano, because that's where basically all of our issues running it stem from 🙏

Copy code

[2022-09-14, 00:01:07 UTC] {pod_manager.py:225} INFO - [2m2022-09-14T00:01:07.338676Z[0m [[32m[1minfo     [0m] [1mEnvironment 'dev' is active[0m
[2022-09-14, 00:05:30 UTC] {pod_manager.py:225} INFO - HTTPSConnectionPool(host='<http://hub.meltano.com|hub.meltano.com>', port=443): Max retries exceeded with url: /meltano/api/v1/plugins/extractors/index (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fc7e8e37940>: Failed to establish a new connection: [Errno 101] Network is unreachable'))
[2022-09-14, 00:05:31 UTC] {pod_manager.py:253} WARNING - Pod meltano-test-snowflake-to-csv-824b14b795e64c9c9f529e21317d7ebd log read interrupted but container base still running
[2022-09-14, 00:05:32 UTC] {pod_manager.py:225} INFO - HTTPSConnectionPool(host='<http://hub.meltano.com|hub.meltano.com>', port=443): Max retries exceeded with url: /meltano/api/v1/plugins/extractors/index (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fc7e8e37940>: Failed to establish a new connection: [Errno 101] Network is unreachable'))
[2022-09-14, 00:05:32 UTC] {pod_manager.py:272} INFO - Pod meltano-test-snowflake-to-csv-824b14b795e64c9c9f529e21317d7ebd has phase Running
[2022-09-14, 00:05:34 UTC] {kubernetes_pod.py:433} INFO - Deleting pod: meltano-test-snowflake-to-csv-824b14b795e64c9c9f529e21317d7ebd
[2022-09-14, 00:05:34 UTC] {taskinstance.py:1889} ERROR - Task failed with exception

martin_morset

09/14/2022, 9:39 AM

Update: The same thing happens on

2.5.0

so I guess it's not a versioning problem. Wonder what is causing this? We run jobs using

meltano run <tap> <loader>

martin_morset

09/14/2022, 10:59 AM

Adding the

--log-level debug

flag, I could see that the problem was rather that someone had added a plugin without the lockfile. This in turn caused meltano to look for plugins externally. I fixed it by running

meltano lock --all

Will Da Silva (Arch)

09/14/2022, 1:55 PM

https://github.com/meltano/meltano/issues/6754

aaronsteers

09/14/2022, 3:31 PM

I was going to suggest

meltano lock --all

- glad that worked! Another user recently mentioned wanting to prevent the update check in

Meltano UI

from pinging PyPi - so @Will Da Silva (Arch) - your newly logged issue can potentially help both of these scenarios. 👍

Open in Slack

Previous Next