Just getting started with Meltano and I'm trying t...
# troubleshootingb
brian_plexico
09/25/2023, 5:02 PMJust getting started with Meltano and I'm trying to connect to an API where there is only one endpoint and all auth information and query params are passed in the body as json. My meltano.yml is:
Copy code
version: 1
default_environment: dev
project_id: 84ee0daa-98cb-442b-ac54-d65f476bfd32
environments:
- name: dev
config:
plugins:
extractors:
- name: tap-rest-api-msdk
config:
api_url: https://<sub>.<domain>.com/default.aspx
streams:
- name: default
- name: staging
- name: prod
plugins:
extractors:
- name: tap-rest-api-msdk
variant: widen
pip_url: tap-rest-api-msdk
config:
api_url: https://<sub>.<domain>.com/default.aspx
params:
credentials:
org: <pass in org value>
username: <pass in username value>
password: <pass in password value>
database: <pass in db name>
use_request_body_not_params: true
streams:
- name: default
loaders:
- name: target-jsonl
variant: andyh1203
pip_url: target-jsonlmeltano config tap-rest-api-msdk testmeltano config--environment=<environment name>l
lukasz_prasol
09/25/2023, 5:15 PMThat's probably copy/paste but there's missing
'curl"b
brian_plexico
09/25/2023, 5:33 PMyeah, the missing
'l
lukasz_prasol
09/25/2023, 5:54 PMError seems related to json - worst case scenario I would go to the file
File "/Users/bplexico/dev/meltano-projects/testetl/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.11/site-packages/requests/models.py", line 971/Users/bplexico/dev/meltano-projects/testetl/.meltano/run/tap-rest-api-msdk/tap.8c88e553-e803-4657-8680-fc124c9b2a64.config.jsons
steve_clarke
09/25/2023, 9:38 PMMay I suggest you look at the requests being generated. If you uncomment the lines https://github.com/s7clarke10/tap-rest-api-msdk/blob/02fdcd916658eafd1a0cd6e26ed161f5c0a8a34e/tap_rest_api_msdk/streams.py#L27-L37 in the venv directory for the tap it will provide additional logging information.
Run your tap with log-level=debug - this will provide more details. I suspect the complexity of the dictionary of parameters sent in the body is perhaps contributing to this issue, it is wanting individual parameters and cannot handle the credentials nested inside the credentials parameter.
It would be interesting if you set the streams via an environment variable like so.
Copy code
export TAP_REST_API_MSDK_STREAMS='[{"name": "default", "params": {"credentials": { "org": "<pass in org value>", "username": "<pass in username>", "password":"<pass in password>" }}, "path": "/", "primary_keys": []}]' Copy code
meltano --log-level=debug invoke tap-rest-api-msdk --discoversteve_clarke
09/26/2023, 4:31 AMLooking at this error further, I believe it doesn't like the response it received back and is erroring when it is trying to parse the JSON while building up schema for the self-discovery.
https://github.com/s7clarke10/tap-rest-api-msdk/blob/02fdcd916658eafd1a0cd6e26ed161f5c0a8a34e/tap_rest_api_msdk/tap.py#L535C36-L535C36
To debug what is going on it would be useful to see the values of
• records_path
• r (response back)
It might be good to also add the http_request debugging lines (which are commented out in the streams.py - see last post) to the beginning of the tap.py program and uncomment them - place them directly the venv (follow the path to the files via the error messages). This will show what is happening with the request and run meltano with a log-level=debug.
I suspect the r.json() is not valid / expected when it is being parsed by line 535. Add these lines before line 535.
Copy code
<http://self.logger.info|self.logger.info>(f"{records_path=}")
<http://self.logger.info|self.logger.info>(f"{r=}")b
brian_plexico
09/26/2023, 4:44 PMAfter uncommenting the lines in streams.py as suggested above, I get the following. It looks like it doesn't handle the nesting of the params well.
Copy code
send: b'GET /Default.aspx?ReturnUrl=%2Fdefault.aspx%2F%3Fcredentials%3Dorg%26credentials%3Dusername%26credentials%3Dpassword&credentials=org&credentials=username&credentials=passwordtap.pydefault.aspxbrian_plexico
09/26/2023, 7:53 PMTo narrow down what is going on, I wrote a script that uses the requests library directly and found that the
data=json= Copy code
r = requests.get(
"https://<sub>.<domain>.com/default.aspx",
json={
"credentials": {"org": "<org_value>", "username": "<username_value>", "password": "<password_value>"},
"database": "<database_value>">,
"type": "<type_value>",
},
)data= Copy code
r = requests.get(
self.config["api_url"] + path,
auth=self.http_auth,
data=params,
headers=headers,
)s
steve_clarke
09/27/2023, 9:21 AMOkay, I understand. The streams.py supports sending params via the json parameter (when use_request_body_not_params: true) where as the tap.py doesn't have that logic built into the schema discovery request because it was assumed authentication would use another means other than params sent via the request body.
Out of a matter of interest if you change line https://github.com/s7clarke10/tap-rest-api-msdk/blob/02fdcd916658eafd1a0cd6e26ed161f5c0a8a34e/tap_rest_api_msdk/tap.py#L531C22-L531C22 from
Copy code
params=params, Copy code
json=params,steve_clarke
09/27/2023, 9:36 AMYou can also send params in the stream as well. See the last open search example in the README.md .
b
brian_plexico
09/27/2023, 3:42 PMI tried the
params=json=3 Views