hey All! I'm extremely new to Meltano. I had a quick call with @taylor a little bit ago to walk thru some issues I was seeing with BigQuery, and he recommended that I join here. After our chat, I setup the tap-github, extractor, and it ran fine to target-bigquery. Then I tried to run a subset of our Facebook Ads data (tap-facebook) to target-jsonl. Works fine. I got overzealous, and tried a tap-facebook to target-bigquery, and got a bunch of error info. Not sure what I'm doing wrong? Any ideas? Errors in thread ...

Can you also share your meltano.yml and the command you ran?

I've removed the Facebook account id for privacy reasons. The commands I ran: • meltano --environment=dev invoke tap-github • meltano run tap-github target-jsonl • meltano run tap-github target-bigquery • meltano invoke target-bigquery //command just hangs

it looks like it could be some malformed data coming from the tap. If you invoke just the tap and redirect it to a file does it look valid?

definitely doesn't look consistent:

That looks how I would expect actually. SCHEMA at the top and several RECORD lines. you can do

cat this-file | meltano invoke target-bigquery

and just stream that data to the target.

I replaced "this-file" with the name of the file I wrote to, and ran that command. Error

RITICAL 'RECORD'
CRITICAL ['Traceback (most recent call last):\n', '  File "/Users/pierre.semaan/SemaforWork/semafor-data-meltano/semafor-data-pipeline/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/target_bigquery/_init_.py", line 129, in main\n    for state in state_iterator:\n', '  File "/Users/pierre.semaan/SemaforWork/semafor-data-meltano/semafor-data-pipeline/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/target_bigquery/process.py", line 54, in process\n    for s in handler.handle_record_message(msg):\n', '  File "/Users/pierre.semaan/SemaforWork/semafor-data-meltano/semafor-data-pipeline/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/target_bigquery/processhandler.py", line 179, in handle_record_message\n    nr = format_record_to_schema(nr, self.bq_schema_dicts[stream])\n', '  File "/Users/pierre.semaan/SemaforWork/semafor-data-meltano/semafor-data-pipeline/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/target_bigquery/schema.py", line 438, in format_record_to_schema\n    record[k] = format_record_to_schema(record[k], bq_schema[k]["fields"])\n', '  File "/Users/pierre.semaan/SemaforWork/semafor-data-meltano/semafor-data-pipeline/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/target_bigquery/schema.py", line 438, in format_record_to_schema\n    record[k] = format_record_to_schema(record[k], bq_schema[k]["fields"])\n', '  File "/Users/pierre.semaan/SemaforWork/semafor-data-meltano/semafor-data-pipeline/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/target_bigquery/schema.py", line 438, in format_record_to_schema\n    record[k] = format_record_to_schema(record[k], bq_schema[k]["fields"])\n', '  File "/Users/pierre.semaan/SemaforWork/semafor-data-meltano/semafor-data-pipeline/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/target_bigquery/schema.py", line 441, in format_record_to_schema\n    record[k] = [conversion_dict[bq_schema[k]["type"]](vi) for vi in v]\n', '  File "/Users/pierre.semaan/SemaforWork/semafor-data-meltano/semafor-data-pipeline/.meltano/loaders/target-bigquery/venv/lib/python3.9/site-packages/target_bigquery/schema.py", line 441, in <listcomp>\n    record[k] = [conversion_dict[bq_schema[k]["type"]](vi) for vi in v]\n', "KeyError: 'RECORD'\n"]

I’ll cross-post in our internal engineering channel to get more eyes on it. I’m at the end of my day and need to hop on Dad duties!

thanks for your help today! Legit greatly appreciated

https://github.com/adswerve/target-bigquery/issues/34 sounds like it might be a related solution!

hmmm. Searching for "tap-catalog.json" inside of my directory, it doesn't exist? Am I missing something?

you would have to dump the catalog using meltano invoke --dump=catalog <plugin>, edit it, then feed it back to the plugin directly https://docs.meltano.com/reference/command-line-interface#invoke

meltano invoke --dump=catalog target-bigquery target-bigquery.json 2023-01-27T162421.245607Z [info ] Environment 'dev' is active Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to join our friendly Slack community. Could not dump catalog: 'catalog'

dump the tap-facebook catalog. the catalog is what’s used to generate the SCHEMA message sent to the target

a couple of questions: • Should I be dumping all Facebook ad streams for this? I'm currently only testing with adscreative • How do I need to feed the json back to the plugin? Something like this?

meltano state set --force dev:tap-facebook-to-target-target-bigquery --input-file tap-facebook.json

Hey @pierre_s 👋 - the file you've dumped is the catalog json which is different from the state json that youre trying to set. You can pass in a custom catalog to your extractor following these docs.

hey Pat, I have no issue jumping into a zoom with you if it makes life easier

cool yeah that might make it easier but let me try inspecting some sample records first to see if theres anything obvious, if that works for you 😄

{"instagram_actor_id": "", "effective_object_story_id": "", "image_hash": "", "title": "", "image_url": "", "object_type": "SHARE", "thumbnail_url": "", "id": "", "account_id": "", "body": "", "name": "", "call_to_action_type": "SIGN_UP", "object_story_spec": {"page_id": "", "instagram_actor_id": "", "link_data": {"link": "", "caption": "", "image_hash": "", "call_to_action": {"type": "SIGN_UP", "value": {"link": ""}}}}, "status": "ACTIVE", "effective_instagram_story_id": "", "actor_id": "", "instagram_permalink_url": ""} {"instagram_actor_id": "", "effective_object_story_id": "", "object_type": "PRIVACY_CHECK_FAIL", "id": "", "account_id": "", "name": "", "call_to_action_type": "SIGN_UP", "object_story_spec": {"page_id": "", "instagram_actor_id": "", "link_data": {"link": "", "message": "", "caption": "", "image_hash": "", "call_to_action": {"type": "SIGN_UP", "value": {"link": ""}}, "image_crops": {"191x100": [[0, 976], [2048, 2048]]}}}, "status": "ACTIVE", "effective_instagram_story_id": "", "actor_id": "", "instagram_permalink_url": ""} {"instagram_actor_id": "", "effective_object_story_id": "", "object_type": "PRIVACY_CHECK_FAIL", "id": "", "account_id": "", "name": "", "call_to_action_type": "SIGN_UP", "object_story_spec": {"page_id": "", "instagram_actor_id": "", "link_data": {"link": "", "message": "", "caption": "", "image_hash": "", "call_to_action": {"type": "SIGN_UP", "value": {"link": ""}}}}, "status": "ACTIVE", "effective_instagram_story_id": "", "actor_id": "", "instagram_permalink_url": ""} {"instagram_actor_id": "", "effective_object_story_id": "", "object_type": "PRIVACY_CHECK_FAIL", "id": "", "account_id": "", "name": "{{product.name}} ", "object_story_spec": {"page_id": "", "instagram_actor_id": ""}, "status": "ACTIVE", "actor_id": ""} {"instagram_actor_id": "", "effective_object_story_id": "", "object_type": "SHARE", "thumbnail_url": "", "id": "", "account_id": "", "name": "{{product.name}} ", "object_story_spec": {"page_id": "", "instagram_actor_id": ""}, "status": "ACTIVE", "actor_id": ""} {"instagram_actor_id": "", "effective_object_story_id": "", "image_hash": "", "title": "", "image_url": "", "object_type": "SHARE", "thumbnail_url": "", "id": "", "account_id": "", "body": "", "name": "", "call_to_action_type": "SIGN_UP", "object_story_spec": {"page_id": "", "instagram_actor_id": "", "link_data": {"link": "", "attachment_style": "link", "call_to_action": {"type": "SIGN_UP"}}}, "status": "ACTIVE", "effective_instagram_story_id": "", "actor_id": "", "instagram_permalink_url": ""} {"instagram_actor_id": "", "effective_object_story_id": "", "image_hash": "", "title": "", "image_url": "", "object_type": "SHARE", "thumbnail_url": "", "id": "", "account_id": "", "body": "", "name": "", "call_to_action_type": "SIGN_UP", "object_story_spec": {"page_id": "", "instagram_actor_id": "", "link_data": {"link": "", "attachment_style": "link", "call_to_action": {"type": "SIGN_UP"}}}, "status": "ACTIVE", "effective_instagram_story_id": "", "actor_id": "", "instagram_permalink_url": ""} {"instagram_actor_id": "", "effective_object_story_id": "", "object_type": "PRIVACY_CHECK_FAIL", "id": "", "account_id": "", "name": "", "call_to_action_type": "LEARN_MORE", "object_story_spec": {"page_id": "", "instagram_actor_id": "", "link_data": {"link": "", "message": "", "name": "", "attachment_style": "link", "description": " ", "call_to_action": {"type": "LEARN_MORE"}}}, "status": "ACTIVE", "effective_instagram_story_id": "", "actor_id": "", "instagram_permalink_url": ""}

I wonder if you add

!adcreative.object_story_spec

or

!adcreative.image_crops

to your selection criteria. It might narrow down what is causing the issue. It does a fair amount of manipulation to these records and seems like its not assigning the right type to one of these object/array fields. Interestingly they have a whole test suite for facebook streams https://github.com/adswerve/target-bigquery/blob/master/tests/rsc/schemas/input_json_schemas_facebook.json

could it be an issue with the extractor being on v14, and our FB Ads account is on 15?

Possibly but I'd guess its unlikely, seems more like an issue with how target-bigquery is handling the records. @ruslan_bergenov (since I know you maintain this target-bigquery variant 😄) is there anything obvious going on here? Do you think its similar to whats happening in https://github.com/adswerve/target-bigquery/issues/34?

I'm getting errors with just pulling from Facebook

yeah until you do an app approval through facebook you'll likely run into rate limiting issues. Once youre approved you get a much higher limit

I'd see if you had time to jump in a zoom, but I'm likely at least an hour away from doing anything with the facebook tap

Hey Pierre - sorry for the delayed response. Pat and I likely won’t be able to hop on an additional call currently. We’ll do what we can async via Slack though - and the community is generally very helpful (though sometimes you may have to ask a time or two).

@pat_nadolny, yes, the issue could be related. The schema / tap-catalog.json may be malformed/invalid. We recommend fixing the schema, like we did in issue 34. Eventually, we plan on addressing issue 35, which might help with similar issues a bit.