https://linen.dev logo
Join Slack
Powered by
Hey everyone can someone help me with `tap-bigquer...
# singer-taps
h
Hey everyone can someone help me with 
tap-bigquery
from meltanolabs: https://github.com/MeltanoLabs/tap-bigquery? Google Analytics saves the events table in bigquery and I would like to extract it from bigquery and save it into postgres by running 
meltano run tap-bigquery target-postgres
. The error I am getting is:
Copy code
google.api_core.exceptions.BadRequest: 400 POST <https://bigquery.googleapis.com/bigquery/v2/projects/salvy-production/queries?prettyPrint=false>: Cannot access field key on a value with type ARRAY<STRUCT<key STRING, value STRUCT<string_value STRING, int_value INT64, float_value FLOAT64, ...>>>
Have anyone faced a similar problem when trying to get data from bigquery? Thanks!
✅ 1
Can’t paste the full stack trace but I noticed that the SELECT statement is quite big, any config that I could use to filter it down?
Copy code
2024-02-28T19:23:50.091367Z [info     ] [SQL: SELECT `events_20240227`.`event_date`, `events_20240227`.`event_timestamp`, `events_20240227`.`event_name`, `events_20240227`.`event_params`, `events_20240227`.`event_params`.`key`, `events_20240227`.`event_params`.`value`, `events_20240227`.`event_params`.`value`.`string_value`, `events_20240227`.`event_params`.`value`.`int_value`, `events_20240227`.`event_params`.`value`.`float_value`, `events_20240227`.`event_params`.`value`.`double_value`, `events_20240227`.`event_previous_timestamp`, `events_20240227`.`event_value_in_usd`, `events_20240227`.`event_bundle_sequence_id`, `events_20240227`.`event_server_timestamp_offset`, `events_20240227`.`user_id`, `events_20240227`.`user_pseudo_id`, `events_20240227`.`privacy_info`, `events_20240227`.`privacy_info`.`analytics_storage`, `events_20240227`.`privacy_info`.`ads_storage`, `events_20240227`.`privacy_info`.`uses_transient_token`, `events_20240227`.`user_properties`, `events_20240227`.`user_properties`.`key`, `events_20240227`.`user_properties`.`value`, `events_20240227`.`user_properties`.`value`.`string_value`, `events_20240227`.`user_properties`.`value`.`int_value`, `events_20240227`.`user_properties`.`value`.`float_value`, `events_20240227`.`user_properties`.`value`.`double_value`, `events_20240227`.`user_properties`.`value`.`set_timestamp_micros`, `events_20240227`.`user_first_touch_timestamp`, `events_20240227`.`user_ltv`, `events_20240227`.`user_ltv`.`revenue`, `events_20240227`.`user_ltv`.`currency`, `events_20240227`.`device`, `events_20240227`.`device`.`category`, `events_20240227`.`device`.`mobile_brand_name`, `events_20240227`.`device`.`mobile_model_name`, `events_20240227`.`device`.`mobile_marketing_name`, `events_20240227`.`device`.`mobile_os_hardware_model`, `events_20240227`.`device`.`operating_system`, `events_20240227`.`device`.`operating_system_version`, `events_20240227`.`device`.`vendor_id`, `events_20240227`.`device`.`advertising_id`, `events_20240227`.`device`.`language`, `events_20240227`.`device`.`is_limited_ad_tracking`, `events_20240227`.`device`.`time_zone_offset_seconds`, `events_20240227`.`device`.`browser`, `events_20240227`.`device`.`browser_version`, `events_20240227`.`device`.`web_info`, `events_20240227`.`device`.`web_info`.`browser`, `events_20240227`.`device`.`web_info`.`browser_version`, `events_20240227`.`device`.`web_info`.`hostname`, `events_20240227`.`geo`, `events_20240227`.`geo`.`city`, `events_20240227`.`geo`.`country`, `events_20240227`.`geo`.`continent`, `events_20240227`.`geo`.`region`, `events_20240227`.`geo`.`sub_continent`, `events_20240227`.`geo`.`metro`, `events_20240227`.`app_info`, `events_20240227`.`app_info`.`id`, `events_20240227`.`app_info`.`version`, `events_20240227`.`app_info`.`install_store`, `events_20240227`.`app_info`.`firebase_app_id`, `events_20240227`.`app_info`.`install_source`, `events_20240227`.`traffic_source`, `events_20240227`.`traffic_source`.`name`, `events_20240227`.`traffic_source`.`medium`, `events_20240227`.`traffic_source`.`source`, `events_20240227`.`stream_id`, `events_20240227`.`platform`, `events_20240227`.`event_dimensions`, `events_20240227`.`event_dimensions`.`hostname`, `events_20240227`.`ecommerce`, `events_20240227`.`ecommerce`.`total_item_quantity`, `events_20240227`.`ecommerce`.`purchase_revenue_in_usd`, `events_20240227`.`ecommerce`.`purchase_revenue`, `events_20240227`.`ecommerce`.`refund_value_in_usd`, `events_20240227`.`ecommerce`.`refund_value`, `events_20240227`.`ecommerce`.`shipping_value_in_usd`, `events_20240227`.`ecommerce`.`shipping_value`, `events_20240227`.`ecommerce`.`tax_value_in_usd`, `events_20240227`.`ecommerce`.`tax_value`, `events_20240227`.`ecommerce`.`unique_items`, `events_20240227`.`ecommerce`.`transaction_id`, `events_20240227`.`items`, `events_20240227`.`items`.`item_id`, `events_20240227`.`items`.`item_name`, `events_20240227`.`items`.`item_brand`, `events_20240227`.`items`.`item_variant`, `events_20240227`.`items`.`item_category`, `events_20240227`.`items`.`item_category2`, `events_20240227`.`items`.`item_category3`, `events_20240227`.`items`.`item_category4`, `events_20240227`.`items`.`item_category5`, `events_20240227`.`items`.`price_in_usd`, `events_20240227`.`items`.`price`, `events_20240227`.`items`.`quantity`, `events_20240227`.`items`.`item_revenue_in_usd`, `events_20240227`.`items`.`item_revenue`, `events_20240227`.`items`.`item_refund_in_usd`, `events_20240227`.`items`.`item_refund`, `events_20240227`.`items`.`coupon`, `events_20240227`.`items`.`affiliation`, `events_20240227`.`items`.`location_id`, `events_20240227`.`items`.`item_list_id`, `events_20240227`.`items`.`item_list_name`, `events_20240227`.`items`.`item_list_index`, `events_20240227`.`items`.`promotion_id`, `events_20240227`.`items`.`promotion_name`, `events_20240227`.`items`.`creative_name`, `events_20240227`.`items`.`creative_slot`, `events_20240227`.`items`.`item_params`, `events_20240227`.`items`.`item_params`.`key`, `events_20240227`.`items`.`item_params`.`value`, `events_20240227`.`items`.`item_params`.`value`.`string_value`, `events_20240227`.`items`.`item_params`.`value`.`int_value`, `events_20240227`.`items`.`item_params`.`value`.`float_value`, `events_20240227`.`items`.`item_params`.`value`.`double_value`, `events_20240227`.`collected_traffic_source`, `events_20240227`.`collected_traffic_source`.`manual_campaign_id`, `events_20240227`.`collected_traffic_source`.`manual_campaign_name`, `events_20240227`.`collected_traffic_source`.`manual_source`, `events_20240227`.`collected_traffic_source`.`manual_medium`, `events_20240227`.`collected_traffic_source`.`manual_term`, `events_20240227`.`collected_traffic_source`.`manual_content`, `events_20240227`.`collected_traffic_source`.`gclid`, `events_20240227`.`collected_traffic_source`.`dclid`, `events_20240227`.`collected_traffic_source`.`srsltid`, `events_20240227`.`is_active_user` cmd_type=elb consumer=False name=tap-bigquery producer=True stdio=stderr string_id=tap-bigquery
2024-02-28T19:23:50.092499Z [info     ] FROM `analytics_321851012`.`events_20240227`] cmd_type=elb consumer=False name=tap-bigquery producer=True stdio=stderr string_id=tap-bigquery
e
h
No luck 😞 this is my meltano.yaml
Copy code
- name: tap-bigquery
      variant: meltanolabs
      pip_url: git+<https://github.com/MeltanoLabs/tap-bigquery.git>
      config:
        project_id: salvy-production
        credentials_path: /secrets/bigquery.json
        select:
          - "analytics_321851012-events_20240227.user_pseudo_id"
          - "analytics_321851012-events_20240227.event_name"
          - "analytics_321851012-events_20240227.event_timestamp"
        stream_maps:
          analytics_321851012-events_20240227:
            __key_properties__:
              - user_pseudo_id
              - event_name
              - event_timestamp
Aha! select is a sibling of config, thanks Edgar it worked after fixing it
👌 1
Hey @Edgar Ramírez (Arch.dev), would you know how to enable incremental replication with this tap? I looked through the config options but couldn’t find an option to set the name for the replication key
e
h
You are the best! Thanks a lot 🙏
np 1
Open in Slack
PreviousNext
Powered by