Hi all, thanks for a great product! Everything is ...
# troubleshooting
m
Hi all, thanks for a great product! Everything is working great locally, and we've now deployed our meltano project on kubernetes where we already had airflow running. However, there is a hiccup: The job hangs after
Environment 'dev' is active
(see screenshot). After about 15 minutes or so, the actual task completes, but the container keeps hanging telling me that the
meltano run
process has not yet exited and is blocked somehow. To me it seems like it is either stuck in an infinite loop, or trying to call something over the network with (seemingly) infinite retries/timeout. We have a strict firewall policy where we explicitly name the domains to be whitelisted for egress out from our environment (required as we are in the telco business with very strict data protection requirements). My theory is that meltano is trying to call out to some external site. I have disabled telemetry in
meltano
in general (MELTANO_SEND_ANONYMOUS_USAGE_STATS="False" as stated in the docs). Any ideas?
As evident in the logs, it hangs for about 6 minutes before getting started.
Update: So the culprit is found. It is trying to contact "sp.meltano.com" with an exponential retry backoff. How can I disable this behavior?
w
Within the project directory, you should find a file
.meltano/analytics.json
. It contains a dictionary. Set the value of the key
send_anonymous_usage_stats
to
false
.
v
Can you verify
MELTANO_SEND_ANONYMOUS_USAGE_STATS
is set run
meltano config meltano --list
it'll give you all the meltano settings that should be one. Then you can set ``MELTANO_SEND_ANONYMOUS_USAGE_STATS` either by a bunch of ways a few of which are by
env
variable or in your
meltano.yml
Seems like a bug that the job would hang if it can't reach
<http://sp.meltano.com|sp.meltano.com>
though?
w
Yes.
v
https://meltano.slack.com/archives/C01TCRBBJD7/p1659100845294559?thread_ts=1659099832.402809&amp;cid=C01TCRBBJD7 interesting stuff! I would just be worried about new Meltano projects not working due to
.meltano
being ephemeral, but for a test that's a good idea!
w
If
.meltano/analytics.json
does not exist (e.g. because it's your first run, or you're re-creating the environment each time) then you can build it manually like so (Python code):
Copy code
import hashlib
import json
import uuid


with open(".meltano/analytics.json", "w") as analytics_json_file:
    json.dump(
        {
            "send_anonymous_usage_stats": False,
            "project_id": str(
                uuid.UUID(hashlib.sha256(b"<your project ID>").hexdigest()[::2])
            ),
            "client_id": str(uuid.uuid4()),  # Any UUID will work for the client ID
        },
        analytics_json_file,
    )
@martin_morset Would you like to file a bug about this yourself, or would you rather I file it on your behalf with the information you've provided here?
m
I can do it, but it'll have to wait until monday.
Thanks for the help! I'll try adding that to the json file dynamically as in your python snippet and report back with results!
@Will Da Silva (Arch) Oups, I was a bit too quick. Seems like that is indeed set to false in the job: I'll file the bug anyways.
w
It would be set to false after running a Meltano command. It needs to be set to false before running
meltano
to fully prevent any requests from being sent.
m
Ah, nice! I'll try that.
w
An easier alternative to updating
analytics.json
as shown above yourself might be to just run some other command before
meltano run
Such as
meltano config --help > /dev/null 2>&1
That would create/update
.meltano/analytics.json
to what we want it to be.
Might need an external timeout, like
timeout --kill-after=3.0 2.0 meltano config --help > /dev/null 2>&1
@martin_morset I've logged the issue: https://github.com/meltano/meltano/issues/6532
@martin_morset Should be fixed in the next release: https://github.com/meltano/meltano/pull/6533
Thank you for the detailed issue report!