has anyone else successfully set up `tap google search conso Meltano #plugins-general

has anyone else successfully set up `tap-google-se...

daniel

08/31/2023, 3:38 PM

has anyone else successfully set up

tap-google-search-console

using https://github.com/singer-io/tap-google-search-console? the tap runs "successfully" but produces no data, also when i run

meltano config tap-google-search-console-sites test

i get this:

Copy code

Plugin configuration is invalid
No RECORD message received

the plugin YAML is in the 🧵

daniel

08/31/2023, 3:39 PM

Copy code

plugins:
  extractors:
    - name: tap-google-search-console
      load_schema: tap_google_search_console
      namespace: tap_google_search_console
      pip_url: git+<https://github.com/singer-io/tap-google-search-console.git@v1.0.0>
      executable: tap-google-search-console
      capabilities:
        - properties
        - discover
        - state
      settings:
        - name: client_id
          kind: password
        - name: client_secret
          kind: password
        - name: refresh_token
          kind: password
        - name: start_date
          kind: date_iso8601
        - name: site_urls
          kind: string
        - name: user_agent
          kind: string
      config:
        client_id: $GOOGLE_SEARCH_CONSOLE_CLIENT_ID
        client_secret: $GOOGLE_SEARCH_CONSOLE_CLIENT_SECRET
        refresh_token: $GOOGLE_SEARCH_CONSOLE_REFRESH_TOKEN
        user_agent: meltano
        start_date: "2022-01-01T00:00:00Z"
        site_urls: "sc-domain:<http://example.com|example.com>"
    - name: tap-google-search-console-sites
      inherit_from: tap-google-search-console
      select:
        - sites.permission_level
        - sites.site_url

edgar_ramirez_mondragon

08/31/2023, 3:54 PM

Hi @daniel. What do you see in the output of

meltano select tap-google-search-console --list --all

daniel

08/31/2023, 3:57 PM

Copy code

Legend:
        selected
        excluded
        automatic

Enabled patterns:
        *.*

Selected attributes:
        [selected ] performance_report_country.clicks
        [automatic] performance_report_country.country
        [selected ] performance_report_country.ctr
        [automatic] performance_report_country.date
        [selected ] performance_report_country.impressions
        [selected ] performance_report_country.position
        [automatic] performance_report_country.search_type
        [automatic] performance_report_country.site_url
        [selected ] performance_report_custom.clicks
        [selected ] performance_report_custom.country
        [selected ] performance_report_custom.ctr
        [automatic] performance_report_custom.date
        [selected ] performance_report_custom.device
        [automatic] performance_report_custom.dimensions_hash_key
        [selected ] performance_report_custom.impressions
        [selected ] performance_report_custom.page
        [selected ] performance_report_custom.position
        [selected ] performance_report_custom.query
        [automatic] performance_report_custom.search_type
        [automatic] performance_report_custom.site_url
        [selected ] performance_report_date.clicks
        [selected ] performance_report_date.ctr
        [automatic] performance_report_date.date
        [selected ] performance_report_date.impressions
        [selected ] performance_report_date.position
        [automatic] performance_report_date.search_type
        [automatic] performance_report_date.site_url
        [selected ] performance_report_device.clicks
        [selected ] performance_report_device.ctr
        [automatic] performance_report_device.date
        [automatic] performance_report_device.device
        [selected ] performance_report_device.impressions
        [selected ] performance_report_device.position
        [automatic] performance_report_device.search_type
        [automatic] performance_report_device.site_url
        [selected ] performance_report_page.clicks
        [selected ] performance_report_page.ctr
        [automatic] performance_report_page.date
        [selected ] performance_report_page.impressions
        [automatic] performance_report_page.page
        [selected ] performance_report_page.position
        [automatic] performance_report_page.search_type
        [automatic] performance_report_page.site_url
        [selected ] performance_report_query.clicks
        [selected ] performance_report_query.ctr
        [automatic] performance_report_query.date
        [selected ] performance_report_query.impressions
        [selected ] performance_report_query.position
        [automatic] performance_report_query.query
        [automatic] performance_report_query.search_type
        [automatic] performance_report_query.site_url
        [selected ] sitemaps.contents
        [selected ] sitemaps.errors
        [selected ] sitemaps.is_pending
        [selected ] sitemaps.is_sitemaps_index
        [selected ] sitemaps.last_downloaded
        [automatic] sitemaps.last_submitted
        [automatic] sitemaps.path
        [automatic] sitemaps.site_url
        [selected ] sitemaps.type
        [selected ] sitemaps.warnings
        [selected ] sites.permission_level
        [automatic] sites.site_url

daniel

08/31/2023, 7:37 PM

FWIW the tap is skipping every stream (even if we're explicitly selecting from them in the Meltano YAML), this same behavior seems to occur in

tap-google-search-console

itself:

Copy code

$ tap-google-search-console --config config.json --catalog catalog.json
INFO Authorized, token expires = 2023-08-31 20:33:47.815405+00:00
INFO Skipping stream: sites
INFO Skipping stream: sitemaps
INFO Skipping stream: performance_report_custom
INFO Skipping stream: performance_report_date
INFO Skipping stream: performance_report_country
INFO Skipping stream: performance_report_device
INFO Skipping stream: performance_report_page
INFO Skipping stream: performance_report_query
{"type": "STATE", "value": {"currently_syncing": null}}

any thoughts @edgar_ramirez_mondragon?

edgar_ramirez_mondragon

08/31/2023, 7:51 PM

FWIW the tap is skipping every stream (even if we're explicitly selecting from them in the Meltano YAML)

Did you confirm this with

meltano invoke tap-google-search-console

daniel

08/31/2023, 8:16 PM

i was using

meltano elt tap-google-search-console target-jsonl

and it said in the log messages every stream was skipped (the same as in the original tap)

daniel

08/31/2023, 8:17 PM

i've noticed if i add

"selected": true

to the

metadata

catalog.json

for the original tap it works, do you know how i can specify this within the

meltano.yaml

edgar_ramirez_mondragon

08/31/2023, 8:19 PM

Oh, that's what

select

is for: https://docs.meltano.com/concepts/plugins/#select-extra

daniel

08/31/2023, 8:25 PM

hmm, not sure why the tap thinks nothing is selected in that case, here is my updated YAML, where every stream gets skipped:

Copy code

plugins:
  extractors:
    - name: tap-google-search-console
      load_schema: tap_google_search_console
      namespace: tap_google_search_console
      pip_url: git+<https://github.com/singer-io/tap-google-search-console.git@v1.0.0>
      executable: tap-google-search-console
      capabilities:
        - properties
        - discover
        - state
      settings:
        - name: client_id
          kind: password
        - name: client_secret
          kind: password
        - name: refresh_token
          kind: password
        - name: start_date
          kind: date_iso8601
        - name: site_urls
          kind: string
        - name: user_agent
          kind: string
      config:
        client_id: $GOOGLE_SEARCH_CONSOLE_CLIENT_ID
        client_secret: $GOOGLE_SEARCH_CONSOLE_CLIENT_SECRET
        refresh_token: $GOOGLE_SEARCH_CONSOLE_REFRESH_TOKEN
        user_agent: meltano
        start_date: "2022-01-01T00:00:00Z"
        site_urls: "sc-domain:<http://example.com|example.com>"
        request_timeout: 300
      metadata:
        "*":
          selected: true
      select:
        - sites.*
        - performance_report_country.*
        - performance_report_custom.*
        - performance_report_date.*
        - performance_report_device.*
        - performance_report_page.*
        - performance_report_query.*
        - sitemaps.*

edgar_ramirez_mondragon

08/31/2023, 8:27 PM

oic what's going on, change capability

properties

for

catalog

daniel

08/31/2023, 8:29 PM

let me try that 🤞

daniel

08/31/2023, 8:49 PM

that seems to have worked, thank you!

edgar_ramirez_mondragon

08/31/2023, 9:00 PM

Awesome! I was curious if we had to add this plugin to the Hub or at least update its capabilities. The plugin is present (https://hub.meltano.com/extractors/tap-google-search-console--singer-io/), and the capabilities and settings seem alright, so going the

meltano add extractor tap-google-search-console --variant=singer-io

route should've worked too.

daniel

08/31/2023, 9:20 PM

one last (unrelated) thing, do you have any familiarity with the Bing Ads tap (https://github.com/singer-io/tap-bing-ads) by chance? that's another data source i need to ingest but the documentation seems quite sparse (and the maintenance status is stale according to MeltanoHub)

edgar_ramirez_mondragon

08/31/2023, 9:51 PM

No familiarity unfortunately. Judging by the old issues and prs it seems that a new effort should be made to fork it and support, or build it from scratch. Another option is to use https://hub.meltano.com/extractors/tap-bing-ads--airbyte/.

Open in Slack

Previous Next