https://linen.dev logo
Join Slack
Powered by
has anyone tried to use a service account with tap...
# plugins-general
n
has anyone tried to use a service account with tap-google-analytics? I have no clue why but it keeps complaining that it can't find an OAuth client in the logs and I just don't know what I can do to fix that other than to try and switch to OAuth credentials instead
d
I set this pipeline up by following the steps on https://meltano.com/plugins/extractors/google-analytics.html months ago, and it's been successfully running daily ever since: https://meltano.meltanodata.com/pipelines/ga-2019
I assume you've tried to regenerate your service account and secrets?
n
yeah, and tried to set it all up over again in a fresh meltano setup. @douwe_maan is there a way to run 
meltano invoke
in a way that gets me the request output from the client? The error message is frustratingly vague for this one so anything you could think of to debug would be helpful
d
@nevin_morgan I'd suggest adding extra debugging log statements to the tap's Python files, which will live inside your project at 
.meltano/extractors/tap-google-analytics/venv/lib/python3.6/site-packages/tap_google_analytics
. Just make sure you write to 
stderr
, not 
stdout
, so that the log messages won't be treated as Singer messages.
Those stderr log messages will show in 
meltano elt
and 
meltano invoke
output
n
that's good to know. I will start jumping through and debugging. Massive help as always @douwe_maan
d
Good luck!
n
@douwe_maan I FOUND THE PROBLEM!
meltano is populating the oauth_credentials object with null keys in the tap.config.json and the check in the tap for which client to try and use is this
Copy code
if 'oauth_credentials' in config:
            return GoogleCredentials(
                access_token=config['oauth_credentials']['access_token'],
                refresh_token=config['oauth_credentials']['refresh_token'],
                client_id=config['oauth_credentials']['client_id'],
                client_secret=config['oauth_credentials']['client_secret'],
                token_expiry=None,  # let the library refresh the token if it is expired
                token_uri="<https://accounts.google.com/o/oauth2/token>",
                user_agent="tap-google-analytics (via <http://singer.io|singer.io>)"
            )
        else:
            return ServiceAccountCredentials.from_json_keyfile_dict(config['client_secrets'], SCOPES)
Where should I file an issue, the tap repo or the meltano one?
d
@nevin_morgan Does the output of 
meltano config tap-google-analytics
also show the null keys? Can you share that (redacted) output?
If Meltano is populating 
tap.config.json
incorrectly, this would be a bug in the Meltano repo
n
Copy code
{
  "view_id": "<REDACTED>",
  "key_file_location": "/client-analytics.json",
  "reports": null,
  "oauth_credentials": {
    "access_token": null,
    "refresh_token": null,
    "client_id": null,
    "client_secret": null
  },
  "start_date": "2020-08-15T00:00:00Z",
  "end_date": "2020-08-20T00:00:00Z"
}
@douwe_maan yup ^
d
@nevin_morgan Interesting. Where does 
meltano config tap-google-analytics list
say those values are coming from? The default? How did you set them? In 
.env
? Using which env vars?
n
I set up a clean install of meltano and only added the tap, then used these variables in .env only
Copy code
export GOOGLE_ANALYTICS_API_CLIENT_SECRETS=""
export GOOGLE_ANALYTICS_API_VIEW_ID=""
export GOOGLE_ANALYTICS_API_START_DATE=""
export GOOGLE_ANALYTICS_API_END_DATE=""
from the config list command
Copy code
view_id [env: GOOGLE_ANALYTICS_API_VIEW_ID] current value: '<REDACTED>' (from `.env`)
key_file_location [env: GOOGLE_ANALYTICS_API_CLIENT_SECRETS] current value: '/client-analytics.json' (from `.env`)
reports [env: GOOGLE_ANALYTICS_API_REPORTS] current value: None (from default)
oauth_credentials.access_token [env: GOOGLE_ANALYTICS_API_OAUTH_ACCESS_TOKEN] current value: None (from default)
oauth_credentials.refresh_token [env: GOOGLE_ANALYTICS_API_OAUTH_REFRESH_TOKEN] current value: None (from default)
oauth_credentials.client_id [env: GOOGLE_ANALYTICS_API_OAUTH_CLIENT_ID] current value: None (from default)
oauth_credentials.client_secret [env: GOOGLE_ANALYTICS_API_OAUTH_CLIENT_SECRET] current value: None (from default)
start_date [env: GOOGLE_ANALYTICS_API_START_DATE] current value: '2020-08-15T00:00:00Z' (from `.env`)
end_date [env: GOOGLE_ANALYTICS_API_END_DATE] current value: '2020-08-20T00:00:00Z' (from `.env`)

Custom:
oauth_credentials [env: TAP_GOOGLE_ANALYTICS_OAUTH_CREDENTIALS] current value: None (from default)
@douwe_maan based on that config does this mean it's a bug on Meltano's side?
d
Ah I finally understand what's happening 🤦‍♂️ It looks like we broke the 
tap-google-analytics
integration with https://gitlab.com/meltano/meltano/-/merge_requests/1802, since we're now always passing an 
oauth_credentials
object, even if it's empty, which is causing 
tap-google-analytics
to go into that 
if 'oauth_credentials' in config:
branch. Instead, Meltano should not be passing the object if it's totally empty, or 
tap-google-analytics
should be verifying 
oauth_credentials
actually has non-null values.
So yeah, this is a bug on Meltano's side. Can you please create an issue?
n
yup will do.
d
@nevin_morgan Can you please verify the fix in https://gitlab.com/meltano/tap-google-analytics/-/merge_requests/11? Just change your 
pip_url
to 
git+<https://gitlab.com/meltano/tap-google-analytics.git@11-with-latest-meltano-fails-with-the-oauth-client-was-not-found-when-oauth_credentials-is-set-but>
and run 
meltano install extractor tap-google-analytics
to install it!
n
@douwe_maan will do
@douwe_maan worked
d
@nevin_morgan Great, thanks! I've merged the MR, so you can restore the 
pip_url
and run 
meltano install extractor tap-google-analytics
again to get it 🙂
Open in Slack
PreviousNext
Powered by