https://linen.dev logo
Join Slack
Powered by
Hi, I seem to need some help :shrug: . I have a B...
# troubleshooting
a
Hi, I seem to need some help 🤷 . I have a BigQuery target that works fine on my dev machine. However when I copy the project to the prod machine, it fails with:
Copy code
2023-04-02T20:02:30.745674Z [info     ] WARNING Compute Engine Metadata server unavailable on attempt 1 of 3. Reason: timed out cmd_type=elb consumer=True name=target-bigquery producer=False stdio=stderr string_id=target-bigquery
2023-04-02T20:02:33.749993Z [info     ] WARNING Compute Engine Metadata server unavailable on attempt 2 of 3. Reason: timed out cmd_type=elb consumer=True name=target-bigquery producer=False stdio=stderr string_id=target-bigquery
2023-04-02T20:02:36.754400Z [info     ] WARNING Compute Engine Metadata server unavailable on attempt 3 of 3. Reason: timed out cmd_type=elb consumer=True name=target-bigquery producer=False stdio=stderr string_id=target-bigquery
2023-04-02T20:02:36.755049Z [info     ] WARNING Authentication failed using Compute Engine authentication due to unavailable metadata server. cmd_type=elb consumer=True name=target-bigquery producer=False stdio=stderr string_id=target-bigquery
2023-04-02T20:02:36.756284Z [info     ] Traceback (most recent call last): cmd_type=elb consumer=True name=target-bigquery producer=False stdio=stderr string_id=target-bigquery

[...]

2023-04-02T20:02:36.760710Z [info     ]     raise exceptions.DefaultCredentialsError(_CLOUD_SDK_MISSING_CREDENTIALS) cmd_type=elb consumer=True name=target-bigquery producer=False stdio=stderr string_id=target-bigquery
2023-04-02T20:02:36.760973Z [info     ] google.auth.exceptions.DefaultCredentialsError: Your default credentials were not found. To set up Application Default Credentials, see <https://cloud.google.com/docs/authentication/external/set-up-adc> for more information. cmd_type=elb consumer=True name=target-bigquery producer=False stdio=stderr string_id=target-bigquery
2023-04-02T20:02:36.885383Z [error    ] Loader failed
2023-04-02T20:02:36.886396Z [error    ] Block run completed.           block_type=ExtractLoadBlocks err=RunnerError('Loader failed') exit_codes={<PluginType.LOADERS: 'loaders'>: 1} set_number=0 success=False
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.
i.e. it doesn't find the metadata server (whatever that is) or my credentials (?) I copied the same 
client_secrets.json
that works on my machine to the same directory (and it has the same access rights) as I use locally. The 
meltano.yml
file is a verbatim copy (bar the project id). On both machines I store the secret JSON with the project and do not specify its path in the environment variable. FWIW • Meltano runs as a Docker container on Ubuntu in both environments. • The tap runs fine with 
target-jsonl
What am I missing? Thanks for any advice!
w
Hi @andras_zimmer. Does your prod machine have network restrictions?
a
Yes, it does. Thanks @Will Da Silva (Arch) for the pointer, it may well be the reason! What output ports does target-bigquery require? (It's somewhat strange to me at this point that I definitely used to use 
bq load
on this machine with the same, existing port restrictions to upload stuff to BigQuery.)
w
it may well be the reason!
Fingers crossed, but it was just a guess 🤷
What output ports does target-bigquery require?
I'm not sure. Maybe @pat_nadolny knows, or knows someone who would know
p
I dont know but @alexander_butler and @Stéphane Burwash are well versed in BQ so maybe they know 🤷
a
I can speak better to my variant, but it seems based on the trace its not using the client_secrets.json and trying to use application default creds instead.
a
Thanks, all of you! thankyou I took a cursory look at the network traffic but did not find anything trivial. I'm not a network guy, though, so it takes a bit more effort on my side. I'll get back when I have time to try @alexander_butler's version and snoop/look around more thoroughly if the network restrictions may really be the cause. (It's likely to be on the order of weeks, though, b/c a lot else is going on.)
@Will Da Silva (Arch) @alexander_butler It took some time (and required a long weekend 😉) for me to return to this. I now have the solution: I had to add the following line to the 
.env
in my project:
Copy code
GOOGLE_APPLICATION_CREDENTIALS='client_secrets.json'
I am not really sure why it helped but it did 🤷. • It's not there in my local machine where it does still work w/o it. • This variable is not in my local environment nor added when calling meltano. (It's used from docker images in both local and prod.) Do note, however, that I have a 
GOOGLE_CLIENT_ID
/ 
GOOGLE_CLIENT_SECRET
pair in my local environment (not in docker but on my local host) for other reasons but they are not there in the production machine. (I temporarily also allowed all outgoing traffic but that did not help. So it was not a network issue -- but that was a reasonable guess anyway.) I hope it helps someone else.
And thanks again!
6 Views
Open in Slack
PreviousNext
Powered by