:speaker: I have an update for all `tap-google-ana...
# plugins-general
p
🔈 I have an update for all
tap-google-analytics
users 🔈. As most of you already know the Universal Analytics API is deprecated and stopped processing new data this month, although my understanding is that historical data will be available for at least another year. With that https://github.com/MeltanoLabs/tap-google-analytics/pull/142 the meltanolabs tap-google-analytics codebase to use the GA4 Data API. Huge shoutout to @connor_flynn @josue_sehnem @Henning Holgersen @Matt Menzenski @alexander_butler for their contributions! How this is reflected on the hub is changing slightly though. When you search the hub you will now see both a GA4 plugin and a UA plugin. The original tap-google-analytics plugin that has been listed for years is now labeled as UA and the meltanolabs variant has its pip_url pinned to a pre-ga4 branch. Theres also a new
tap-ga4
plugin page that represents the GA4 variants. The meltanolabs tap-ga4 variant is still tied to the tap-google-analytics repository but is listed on the hub with the
tap-ga4
alias so with meltano you'd run it using
meltano invoke tap-ga4
. ⚠️ Action Required ⚠️ - If you're using an unpinned pip_url and want to continue using UA you need to pin your pip_url to
main_deprecated_ua
as shown in https://github.com/meltano/hub/blob/1c58814f48c7f08d23d179b6c17076efea0353f2/_data/meltano/extractors/tap-google-analytics/meltanolabs.yml#L18. If you want to switch to GA4 you will need to
meltano add
the new
tap-ga4
plugin and configure it as some settings have changed.
e
Love this! Great to see our old work (especially @connor_flynn's) being utilized
m
I would like to better understand how to use it. I see that a list of reports defined here. But what if I'd like to use premade (defaults) report for user acquisition as it is defined in singer tap. Should I just add the report to defaults/default_report_definition.json? I tried that, but when I execute `meltano run..`:
Copy code
Skipping deselected stream 'user_acq_first_user_default_channel_group_report'
Also, when I execute
meltano select tap-ga4 --list --all
I don't see this
user_acq_first_user_default_channel_group_report
. Only the defaults that are initially created. What do I miss and fail to understand?
p
@mindaugas_nizauskas the tap comes with a default set of reports but you can customize them or create your own using the reports setting
Also, when I execute
meltano select tap-ga4 --list --all
I don't see this...
I wonder if this is related to the catalog file being cached. You could try deleting your
.meltano/run/tap-ga4/
directory to clear out at artifacts that might be getting cached. Then running the
--list --all
command should show your streams and you can select them as needed.
m
Got it. Under reports in my meltano.yml I should put the entire array similarly as it is done for default in the .json file? Mind the syntax of course.
p
Its actually a path to a custom report file. It looks like the UA hub page has better docs on that setting https://hub.meltano.com/extractors/tap-google-analytics#reports-setting. I'll take a note to get the GA4 page updated to include more details
d
that is good to know, thank you! I noticed that I was able easily to run the Google Reporting API and for Data API I am always getting:
Copy code
KeyError: 'refresh_token'
I did all the same to obtain Access and Refresh token. But it is not working. What is the right way to get tokens?
Copy code
The error message suggests that the Google Analytics 4 (GA4) tap in Meltano is unable to find the refresh_token in the configuration file.

The refresh_token is required for Meltano to gain authorization to your Google Analytics data. This token is usually obtained through the OAuth2 process. After you get an access token, you can use it to access Google APIs. When the access token expires, the application uses the refresh token to get a new one.

Based on the error, it seems like you've not properly set up the OAuth2 process or that the refresh_token is not included in the configuration file for the tap-ga4 plugin.
but I see a value:
Copy code
oauth_credentials.refresh_token [env: TAP_GA4_OAUTH_CREDENTIALS_REFRESH_TOKEN] current value:
I found in old tap:
Copy code
def _initialize_credentials(self):
        if self.config.get("oauth_credentials"):
            return GoogleCredentials(
                access_token=self.config["oauth_credentials"]["access_token"],
                refresh_token=self.config["oauth_credentials"]["refresh_token"],
                client_id=self.config["oauth_credentials"]["client_id"],
                client_secret=self.config["oauth_credentials"]["client_secret"],
                token_expiry=None,  # let the library refresh the token if it is expired
                token_uri="<https://accounts.google.com/o/oauth2/token>",
                user_agent="tap-google-analytics (via <http://singer.io|singer.io>)",
            )
but in new:
Copy code
def _initialize_credentials(self):
        if self.config.get("oauth_credentials"):
            return OAuthCredentials(
                None,
                refresh_token=self.config["refresh_token"],
                client_id=self.config["client_id"],
                client_secret=self.config["client_secret"],
                token_uri="<https://accounts.google.com/o/oauth2/token>",
            )
why
tap-ga4
doesn't have the
access_token
? I see the method is changed to from
GoogleCredentials
to
Copy code
from google.oauth2.service_account import Credentials as OAuthCredentials
And If I check what kind arguments it has:
Copy code
self,
        signer,
        service_account_email,
        token_uri,
        scopes=None,
        default_scopes=None,
        subject=None,
        project_id=None,
        quota_project_id=None,
        additional_claims=None,
        always_use_jwt_access=False,
        universe_domain=_DEFAULT_UNIVERSE_DOMAIN,
        trust_boundary=None,
I don't see refresh, access, client, secret.
it is something might be wrong for this kind of configuration:
Copy code
oauth_credentials.access_token
oauth_credentials.client_id
oauth_credentials.client_secret
oauth_credentials.refresh_token
it is not using the
refresh_token
p
@dima_anoshin thanks for raising this! You make good points, our tests are setup to use the service account and not the oauth configurations so this wasnt caught. I created a draft PR that somewhat reverts to the old auth method but I'm not able to test it easily. At this point its basically a guess at how to support it because I couldnt find any good docs on this, have you? If you have a chance you can test it by updating your pip url to
git+<https://github.com/MeltanoLabs/tap-google-analytics.git@fix_oauth_creds>
and running
meltano install extractor tap-ga4 --clean
.
Update: I was able to test this and validate that it works. I'll keep you posted when it merges
d
Appreciate the quick help here!
works great now! Do you have any suggestion on Rate Limit when extract data from GA?