Hi! [meltano newbie here] I'm having an issue run...
# troubleshooting
c
Hi! [meltano newbie here] I'm having an issue running
meltano select tap-rest-api-msdk --list
getting the following error. I've been able to
meltano config tap-rest-api-msdk
so far. Not entirely sure if I missed any step? Maybe I need to configure the streams first? thinkspin ```❯ meltano select tap-rest-api-msdk --list 2023-10-23T105817.110159Z [info ] The default environment 'dev' will be ignored for
meltano select
. To configure a specific environment, please use the option
--environment=<environment name>
. Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to join our friendly Slack community. Cannot list the selected attributes: Catalog discovery failed: command ['/Users/cristianprigoana/Work/meltano-projects/webz-downloader/.meltano/extractors/tap-rest-api-msdk/venv/bin/tap-rest-api-msdk', '--config', '/Users/cristianprigoana/Work/meltano-projects/webz-downloader/.meltano/run/tap-rest-api-msdk/tap.61c6bf15-afb5-4c34-84ae-f859207329dc.config.json', '--discover'] returned 1 with stderr: Traceback (most recent call last): File "/Users/cristianprigoana/Work/meltano-projects/webz-downloader/.meltano/extractors/tap-rest-api-msdk/venv/bin/tap-rest-api-msdk", line 8, in <module> sys.exit(TapRestApiMsdk.cli()) ^^^^^^^^^^^^^^^^^^^^ File "/Users/cristianprigoana/Work/meltano-projects/webz-downloader/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.11/site-packages/click/core.py", line 1157, in call return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/cristianprigoana/Work/meltano-projects/webz-downloader/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.11/site-packages/click/core.py", line 1077, in main with self.make_context(prog_name, args, **extra) as ctx: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/cristianprigoana/Work/meltano-projects/webz-downloader/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.11/site-packages/click/core.py", line 943, in make_context self.parse_args(ctx, args) File "/Users/cristianprigoana/Work/meltano-projects/webz-downloader/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.11/site-packages/click/core.py", line 1408, in parse_args value, args = param.handle_parse_result(ctx, opts, args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/cristianprigoana/Work/meltano-projects/webz-downloader/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.11/site-packages/click/core.py", line 2400, in handle_parse_result value = self.process_value(ctx, value) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/cristianprigoana/Work/meltano-projects/webz-downloader/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.11/site-packages/click/core.py", line 2362, in process_value value = self.callback(ctx, self, value) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/cristianprigoana/Work/meltano-projects/webz-downloader/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 519, in cb_discover tap = cls( ^^^^ File "/Users/cristianprigoana/Work/meltano-projects/webz-downloader/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 104, in init self.mapper.register_raw_streams_from_catalog(self.catalog) ^^^^^^^^^^^^ File "/Users/cristianprigoana/Work/meltano-projects/webz-downloader/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 167, in catalog self._catalog = self.input_catalog or self._singer_catalog ^^^^^^^^^^^^^^^^^^^^ File "/Users/cristianprigoana/Work/meltano-projects/webz-downloader/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 315, in _singer_ca…
r
Probably the tap requires the
streams
setting to be configured: https://hub.meltano.com/extractors/tap-rest-api-msdk#streams-setting
c
interesting, now I see this under my
tap-rest-api-msdk
extractor in
metalon.yml
Copy code
select:
    - '*.*'
Thanks @Reuben (Matatika), I think that helped make some progress. In Github README it does show up explicitly as
required
Now running into a permissions issue. Looks like the tap is sending an API request, but not entirely sure to what URL path. The
api_url
seems to be correct
Copy code
❯ meltano select tap-rest-api-msdk --list
2023-10-23T11:53:53.021041Z [info     ] The default environment 'dev' will be ignored for `meltano select`. To configure a specific environment, please use the option `--environment=<environment name>`.
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Cannot list the selected attributes: Catalog discovery failed: command ['/Users/cristianprigoana/Work/meltano-projects/webz-downloader/.meltano/extractors/tap-rest-api-msdk/venv/bin/tap-rest-api-msdk', '--config', '/Users/cristianprigoana/Work/meltano-projects/webz-downloader/.meltano/run/tap-rest-api-msdk/tap.af70a3d3-f0b6-494b-bcec-f3befcd40c9b.config.json', '--discover'] returned 1 with stderr:
 2023-10-23 12:53:57,336 | INFO     | tap-rest-api-msdk    | No schema found. Inferring schema from API call.
2023-10-23 12:53:57,614 | ERROR    | tap-rest-api-msdk    | Error Connecting, message = <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access this resource.</p>
<hr> 
...
I guess I have to provide the schema myself, instead of letting the tap try to infer it
s
Are you sure you have provided the required credentials for the API, the message seems to indicate that it cannot authenticate. Look at the available auth settings in the README on Github.
c
@steve_clarke thanks for your input! This specific API does not have proper authentication, only a
token
URL param, hence the reason I haven't used any of the authentication settings and configured the
token
in the
params
section