Hello all, i have added all the required config fo...
# plugins-generalm
mohsin
03/04/2021, 10:42 PMHello all, i have added all the required config for my tap-bing-ads, but still facing this issue, the same config works in singer tap.. i also revert the version of bingads but still face this issue..
tap-bing-ads -> target-postgres
Target properly configured..
Copy code
bingads.exceptions.OAuthTokenRequestException: error_code: invalid_grant, error_description: The request was denied because one or more scopes requested are unauthorized or expired. The user must first sign in and grant the client application access to the requested scope.d
douwe_maan
03/04/2021, 10:48 PM@mohsin Can you show the configuration Meltano is using by running
meltano config tap-bing-adsm
mohsin
03/04/2021, 10:50 PM Copy code
{
"developer_token": "*****",
"oauth_client_id": "******8",
"oauth_client_secret": "******",
"refresh_token": **************",
"customer_id": "******",
"account_ids": "********",
"start_date": "2021-03-05T00:00:00Z"
}mohsin
03/04/2021, 10:51 PM@douwe_maan here is the output after the config command
mohsin
03/04/2021, 10:51 PMis required_live_connect required?
d
douwe_maan
03/04/2021, 10:54 PMIt defaults to
Truedouwe_maan
03/04/2021, 10:54 PMIf you run the tap directly with that output as the config file, are you not hitting the same error? Or did you mean something else with "the same config works in singer tap"?
douwe_maan
03/04/2021, 10:56 PMNote also that we are planning to make https://github.com/wheelhousedev/tap-bing-ads the new default fork for tap-bing-ads (https://gitlab.com/meltano/meltano/-/issues/2617), so you may want to switch to that one already. It doesn't look related to this issue, though.
m
mohsin
03/04/2021, 10:57 PMi applied the same config work fine in singer tap, when i remove the require_live_connect and redirect_uri so same error through but added these params, it works fine
mohsin
03/04/2021, 11:00 PMwhat should i have to do?
d
douwe_maan
03/04/2021, 11:00 PMSo the exact output of
meltano config tap-bing-adstap-bing-ads--configdouwe_maan
03/04/2021, 11:00 PMIf you like, I can jump on a quick Zoom call with you to see the situation for myself, that may help me spot what's different!
m
mohsin
03/04/2021, 11:01 PMyeah sure we can do
d
douwe_maan
03/04/2021, 11:23 PM@aditya_sastry It looks like Mohsin is having trouble generating valid tokens/credentials for Bing Ads API. Do you happen to remember the process you followed that worked?
douwe_maan
03/04/2021, 11:25 PMDo you happen to remember which scopes you had to request, since that appears to be the issue according to the error message?
a
aditya_sastry
03/05/2021, 1:12 AMThis one worked fairly well with the following config. I'm planning to switch the endpoint on the wheelhousdev repo from Microsoft Live to IDP when I get some engineering capacity which should help with this.
1. You definitely want
require_live_connect: falsetrued
douwe_maan
03/05/2021, 1:32 AM@aditya_sastry thanks! We should get this documented, I’ll make a note of that
m
mohsin
03/05/2021, 2:06 PM@aditya_sastry Thanks for sharing your words, i will revert r`equire_live_connect: true to false` , second do you have any idea why bing endpoint won’t return the refresh token
When i hit this endpoint: https://login.microsoftonline.com/common/oauth2/v2.0/token will face the below error,
Copy code
"error_description": "AADSTS70000: The provided value for the 'code' parameter is not valid. The code has expired.\r\nTrace ID: c48a954a-3f49-459a-bd49-3478995d6300\r\nCorrelation ID: a6b615f7-116a-4587-bfa0-93e89a51545e\r\nTimestamp: 2021-03-04 23:12:00Z",mohsin
03/05/2021, 2:07 PM@aditya_sastry do we have need to specify the scope in our meltano.yaml file?
mohsin
03/05/2021, 2:07 PMYeah sure we do a call, please let me know when you will be available
mohsin
03/05/2021, 5:42 PM@douwe_maan did you get any info?
a
aditya_sastry
03/05/2021, 6:28 PMYou need the scope in the refresh token request, not the meltano.yml^
https://docs.microsoft.com/en-us/advertising/guides/get-started?view=bingads-13
m
mohsin
03/05/2021, 6:30 PMyeah yeah i am not specifying the scope in meltano.yml
d
douwe_maan
03/05/2021, 10:30 PMI've created a new issue about the incomplete docs: https://gitlab.com/meltano/meltano/-/issues/2630
t
taylor
04/06/2021, 8:40 PMI’ve made a quick MR to update the docs based on my understanding of this thread. @aditya_sastry or @mohsin can you review when you get a chance? https://gitlab.com/meltano/meltano/-/merge_requests/2092/diffs
m
mohsin
04/06/2021, 9:06 PM@taylor sure i’ll look at it
mohsin
04/06/2021, 9:09 PM@taylor looks fine now. Thanks for the update
t
taylor
04/06/2021, 9:10 PMthanks for checking!
m
mohsin
04/07/2021, 6:47 PM@taylor can you please the requireliveconnect to
FalsefalseTruemohsin
04/07/2021, 7:06 PMcheckout this i set my
requireLiveconnectrequireLiveconnect Copy code
bingads.exceptions.OAuthTokenRequestException: error_code: invalid_grant, error_description: The request was denied because one or more scopes requested are unauthorized or expired. The user must first sign in and grant the client application access to the requested scope.mohsin
04/07/2021, 7:07 PMis this error from meltano side or bing ads ?
mohsin
04/07/2021, 7:07 PMcc: @edward_ryan @gunnar
t
taylor
04/07/2021, 7:30 PM@mohsin I’m not sure I understand your request about
requireliveconnectrequire_live_connectm
mohsin
04/07/2021, 7:41 PMyes you updated correctly but unfortunately this will not create any impact if we set
require_live_connectt
taylor
04/07/2021, 7:44 PMoh looks like that’s b/c they’re passing in
REQUIRED_CONFIG_KEYStaylor
04/07/2021, 7:44 PMthat’s annoying.
taylor
04/07/2021, 7:46 PMso the options here areto make a PR to the tap to get that updated or fork the project 😕
m
mohsin
04/07/2021, 9:11 PMyes it is annonying we have to manually added the require_live_connect value into the package, this is not the correct approach can you please guide me how can i raise a request on this?
mohsin
04/07/2021, 9:14 PM1. ad_extension_detail_report.*
2. campaign_performance_report.*
Also these two streams through incremental state error
```target-redshift (out) | {"bookmarks": {"accounts": {"last_record": "2021-02-23T212900.960000+00:00"}, "141559394_ad_extension_detail_report": {"request_id": null, "date": "2021-03-19T000000+00:00"}, "141559394_ad_performance_report": {"request_id": "1238134853920"}, "141559394_age_gender_audience_report": {"request_id": "1238134853947"}, "141559394_audience_performance_report": {"request_id": "1238134853965"}, "141559394_keyword_performance_report": {"request_id": "1238134853997"}}}
meltano | INFO Incremental state has been updated at 2021-03-19 220602.556205.
meltano | DEBUG Incremental state: {'bookmarks': {'accounts': {'last_record': '2021-02-23T212900.960000+00:00'}, '141559394_ad_extension_detail_report': {'request_id': None, 'date': '2021-03-19T000000+00:00'}, '141559394_ad_performance_report': {'request_id': '1238134853920'}, '141559394_age_gender_audience_report': {'request_id': '1238134853947'}, '141559394_audience_performance_report': {'request_id': '1238134853965'}, '141559394_keyword_performance_report': {'request_id': '1238134853997'}}}
meltano | DEBUG Deleted configuration at /home/ubuntu/dev-meltano/bing-to-postgres/meltan-bing-to-redshift/bing-ads-to-redshift/.meltano/run/elt/2021-03-19T220443--tap-bing-ads--target-redshift/4aec8e37-5ffb-4dbf-80b0-1085cf559bee/target.config.json
meltano | DEBUG Deleted configuration at /home/ubuntu/dev-meltano/bing-to-postgres/meltan-bing-to-redshift/bing-ads-to-redshift/.meltano/run/elt/2021-03-19T220443--tap-bing-ads--target-redshift/4aec8e37-5ffb-4dbf-80b0-1085cf559bee/tap.config.json
meltano | ERROR Extraction failed (1): }
meltano | DEBUG ELT could not be completed: Extractor failed
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.8/site-packages/meltano/cli/elt.py", line 236, in _run_elt
await _run_extract_load(elt_context, output_logger)
File "/home/ubuntu/.local/lib/python3.8/site-packages/meltano/cli/elt.py", line 274, in _run_extract_load
await singer_runner.run(
File "/home/ubuntu/.local/lib/python3.8/site-packages/meltano/core/runner/singer.py", line 253, in run
await self.invoke(
File "/home/ubuntu/.local/lib/python3.8/site-packages/meltano/core/runner/singer.py", line 227, in invoke
raise RunnerError("Extractor failed", {PluginType.EXTRACTORS: tap_code})
meltano.core.runner.RunnerError: Extractor failed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.8/site-packages/meltano/cli/elt.py", line 224, in _redirect_output
yield
File "/home/ubuntu/.local/lib/python3.8/site-packages/meltano/cli/elt.py", line 245, in _run_elt
raise CliError(f"ELT could not be completed: {err}") from err
meltano.cli.utils.CliError: ELT could not be completed: Extractor failed
meltano | ELT could not be completed: Extractor failed
[2021-03-19 220602,667] [38353|MainThread|meltano.cli.utils] [DEBUG] ELT could not be completed: Extractor failed
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.8/site-packages/meltano/cli/elt.py", line 236, in _run_elt
await _run_extract_load(elt_context, output_logger)
File "/home/ubuntu/.local/lib/python3.8/site-packages/meltano/cli/elt.py", line 274, in _run_extract_load
await singer_runner.run(
File "/home/ubuntu/.local/lib/python3.8/site-packages/meltano/core/runner/singer.py", line 253, in run
await self.invoke(
File "/home/ubuntu/.local/lib/python3.8/site-packages/meltano/core/runner/singer.py", line 227, in invoke
raise RunnerError("Extractor failed", {PluginType.EXTRACTORS: tap_code})
meltano.core.runner.RunnerError: Extractor failed
The above exception was the direct cause of the following exception:
t
taylor
04/07/2021, 9:19 PMguide me how can i raise a request on thisyou can make a pull request on GitHub to update https://github.com/singer-io/tap-bing-ads/blob/master/tap_bing_ads/__init__.py#L27-L35 to include the
require_live_connecttaylor
04/07/2021, 9:20 PMBased on the errors you’ve shared I’m not sure where the failure would be coming from. Is there any failure earlier in the log output? It’s saying the extractor failed so maybe a specific stream failed but everything else succeeded?
m
mohsin
04/07/2021, 9:21 PMyeah i think stream failed