Hello folks I m trying to load data from tap facebook to tar Meltano #troubleshooting

Hello folks, I'm trying to load data from tap-face...

akmal

09/13/2023, 8:09 AM

Hello folks, I'm trying to load data from tap-facebook to target-bigquery using meltano run tap-facebook target-bigquery. When i invoke tap-facebook, there's no problem. But when i run target-bigquery I got error in during load job to bigquery

Copy code

Error while reading data, error message: JSON parsing error in row starting at position 0: Repeated field must be imported as a JSON array. Field: unique_actions.

and upon looking at the record processed, seems like all json array type are converted to string causing the error. Anyone have idea how to resolve this?

akmal

09/13/2023, 8:29 AM

target-bigquery

config:

Copy code

{
  "batch_size": 500,
  "cluster_on_key_properties": false,
  "column_name_transforms": {
    "add_underscore_when_invalid": false,
    "lower": false,
    "quote": false,
    "snake_case": false
  },
  "credentials_path": ".config/cred.json",
  "dataset": "meltano_ingestion",
  "dedupe_before_upsert": true,
  "denormalized": true,
  "fail_fast": true,
  "flattening_enabled": true,
  "flattening_max_depth": 2,
  "generate_view": false,
  "location": "asia-southeast1",
  "method": "batch_job",
  "options": {
    "process_pool": false,
    "storage_write_batch_mode": false
  },
  "overwrite": false,
  "partition_granularity": "month",
  "project": "",
  "schema_resolver_version": 1,
  "timeout": 600,
  "upsert": true
}

akmal

09/13/2023, 8:48 AM

this is the sample record i print from

target-bigquery

Copy code

{'actions': '[{"action_destination": "696168159220568", "action_target_id": "696168159220568", "action_type": "link_click", "value": 1.0, "1d_click": 1.0, "7d_click": 1.0, "28d_click": 1.0}, {"action_destination": "unknown", "action_target_id": "0", "action_type": "landing_page_view", "value": 2.0, "1d_click": 2.0, "7d_click": 2.0, "28d_click": 2.0}]'}

edgar_ramirez_mondragon

09/13/2023, 4:07 PM

Can you increase the flattening depth? I think that should fix the issue. This is fixed upstream but hasn't been released yet (https://github.com/meltano/sdk/issues/1400)

akmal

09/14/2023, 12:03 PM

Yeah that works! Thanks so much @edgar_ramirez_mondragon! I spent so much time trying to figure out the problem 😂

8 Views

Open in Slack

Previous Next