Hey all. Did some further testing over the weekend...
# troubleshootingp
pierre_s
01/30/2023, 1:57 PMHey all. Did some further testing over the weekend, implementing some of the suggestions from @taylor and @pat_nadolny. My meltano.yml now looks like
plugins:extractors:- name: tap-facebookcatalog: extract/tap-facebook.jsonvariant: singer-iopip_url: git+<https://github.com/singer-io/tap-facebook.git>config:account_id: ''start_date: 01/24/2023end_date: 01/26/2023insights_buffer_days: 0include_deleted: trueselect:- adcreative.*{"filters": {"type": ["null","array"],"items": {"type": ["null","string"]}},"streams": [{"stream": "adcreative","tap_stream_id": "adcreative","schema": {"properties": {"body": {"type": ["null","string"]meltano invoke tap-facebookt
taylor
01/30/2023, 10:11 PMit looks like it’s a bug with the tap. It’s attempting to lowercase a boolean and it bombs on that.
taylor
01/30/2023, 10:13 PMit’s possible thought that by telling meltano to treat it as a string then it would fix it. if you add:
Copy code
settings:
- name: include_deleted
kind: stringp
pierre_s
01/30/2023, 10:27 PMSo, I have a question. My extract/tap-facebook.json has all stream entities and attributes from facebook ads. This seems to be superceding what I define in the meltano.yml, as it pulls adcreative, even though I only want the ads.* stuff ... Did I implement the settings change correctly?
Copy code
INFO Syncing adcreative, fields {'id'}
INFO Syncing ads, fields {'id', 'updated_time'}
INFO METRIC: {"type": "counter", "metric": "record_count", "value": 0, "tags": {"endpoint": "ads"}}
ERROR 'bool' object has no attribute 'lower'
Traceback (most recent call last):t
taylor
01/30/2023, 10:30 PMyou would still need the
include_deleted: truesettingconfigp
pat_nadolny
01/31/2023, 4:44 PMOne other thing to consider is using an alternative bigquery target to see if that helps. @alexander_butler built a pretty robust SDK based target-bigquery intending to fix some of these types of issues. He's been using it in production for a while but we just put it up on MeltanoHub a few days ago
p
pierre_s
01/31/2023, 4:46 PMAppreciate that. Even with your suggestions, I'm still not able to get past the "
Copy code
NFO METRIC: {"type": "counter", "metric": "record_count", "value": 0, "tags": {"endpoint": "ads"}}
ERROR 'bool' object has no attribute 'lower'"p
pat_nadolny
01/31/2023, 5:10 PMahh ok so it looks like the tap is expecting a string version of the boolean like Taylor mentioned. I can override it in the meltano hub definition so its fixed for everyone https://github.com/meltano/hub/pull/1142. If you run
meltano lock tap-facebook --updatepat_nadolny
01/31/2023, 5:10 PMI'm curious if that will help
p
pierre_s
01/31/2023, 5:12 PMis it an issue for everyone? I don't want to break it for everyone. We're locked to V15 of Facebook because we are a relatively new app - our site launched on October 28th, so we can't revert to version 14 and test that, if that makes sense?
p
pat_nadolny
01/31/2023, 5:15 PMI confirmed that the tap is expecting a string version of a boolean and the meltano definition was casting it to a boolean, so anyone trying to use that setting would have the same issue. Its likely that most people just accept the default so they never run into this issue but not sure 🤔
p
pierre_s
01/31/2023, 5:38 PMso maybe I'm missing something, but even if "include_deleted" is not there in the meltano.yml, running
meltano invoke tap-facebook > fb_ads.txtpierre_s
01/31/2023, 5:38 PMalso, ran "meltano lock tap-facebook --update" and it's still an issue. Maybe I've incorrectly defined something
pierre_s
01/31/2023, 5:40 PMthis is my current tap state for dev:
- name: tap-facebookvariant: singer-iopip_url: git+<https://github.com/singer-io/tap-fpacebook.git>settings:- name: include_deletedkind: stringconfig:account_id: ''start_date: 01/24/2023end_date: 01/26/2023insights_buffer_days: 0include_deleted: truecatalog: extract/tap-facebook.jsonselect:- ads.*p
pat_nadolny
01/31/2023, 5:43 PMif your lock command worked then you should be able to remove the
settings:meltano config tap-facebookp
pierre_s
01/31/2023, 5:44 PM Copy code
{
"account_id": "",
"access_token": "",
"start_date": "01/24/2023",
"end_date": "01/26/2023",
"insights_buffer_days": 0,
"include_deleted": true
}pierre_s
01/31/2023, 5:46 PMbut now I'm in timeout from Facebook polling for 90-ish minutes! 🤣
p
pat_nadolny
01/31/2023, 5:50 PMahh ok so you'll need to update the
include_deleted: trueinclude_deleted: 'true'pat_nadolny
01/31/2023, 5:50 PMhahah but yeah the facebook rate limits are a whole other beast
p
pierre_s
01/31/2023, 5:51 PMI asked my boss to do it. We're still running on the principle of least privilege, which is fine, but still a process
pierre_s
01/31/2023, 5:53 PMso, my exported catalog json has all entities and attributes. Is that why if I have this
select:- ads.*p
pat_nadolny
01/31/2023, 5:57 PMcheck out the docs on catalogs https://docs.meltano.com/guide/integration#extractor-catalog-generation. That should clarify whats going on 😄
p
pierre_s
01/31/2023, 6:27 PMso, it's ignroing my inclusion patter of
ads.**,If no rules are defined using meltano select, Meltano will fall back on catch-all rule *.* so that all entities and attributes are selected.2 Views