Hello, I am trying to setup the rest-api tap. But, there seems to be a problem either with auth or my api url. Is there anyway to see which requests are beeing run by the plugin during config test (config tap-rest-api-msdk test)?

meltano --log-level debug config tap-rest-api-msdk test

might reveal something.

Thank you for the quick reply. debug logging gives me more output. Apparently the discover step fails since the API cannot be reached (404). But I am sure the configured endpoint does exist. Unfortunately the requested uri is still not logged with debug logging. The request seems to happen somewhere in the plugin after --discover is beeing called. If I run discover explicitly I get this logoutput:

./meltano --log-level debug invoke tap-rest-api-msdk --discover
2025-05-12T11:39:57.842438Z [debug    ] Meltano 3.7.5, Python 3.9.22, Linux (x86_64)
2025-05-12T11:39:57.845270Z [debug    ] /etc/timezone found, contents:
 Etc/UTC

2025-05-12T11:39:57.845678Z [debug    ] /etc/localtime found
2025-05-12T11:39:57.846301Z [debug    ] 2 found:
 {'/etc/timezone': 'Etc/UTC', '/etc/localtime is a symlink to': 'Etc/UTC'}
2025-05-12T11:39:57.849381Z [info     ] Environment 'dev' is active
2025-05-12T11:39:57.869161Z [debug    ] Creating DB engine for project at '/projects' with DB URI 'sqlite:////projects/.meltano/meltano.db'
2025-05-12T11:39:57.952781Z [debug    ] Found plugin parent            parent=tap-rest-api-msdk plugin=tap-rest-api-msdk source=LOCKFILE
2025-05-12T11:39:58.130256Z [debug    ] Skipped installing extractor 'tap-rest-api-msdk'
2025-05-12T11:39:58.130593Z [debug    ] Skipped installing 1/1 plugins
2025-05-12T11:39:58.543236Z [debug    ] Created configuration at /projects/.meltano/run/tap-rest-api-msdk/tap.058892ab-dba7-4bae-8dd4-14075e767e45.config.json
2025-05-12T11:39:58.543581Z [debug    ] Could not find tap.properties.json in /projects/.meltano/extractors/tap-rest-api-msdk/tap.properties.json, skipping.
2025-05-12T11:39:58.543782Z [debug    ] Could not find tap.properties.cache_key in /projects/.meltano/extractors/tap-rest-api-msdk/tap.properties.cache_key, skipping.
2025-05-12T11:39:58.543918Z [debug    ] Could not find state.json in /projects/.meltano/extractors/tap-rest-api-msdk/state.json, skipping.
2025-05-12T11:39:58.544087Z [debug    ] Could not find tap.singer_sdk_logging.json in /projects/.meltano/extractors/tap-rest-api-msdk/tap.singer_sdk_logging.json, skipping.
2025-05-12T11:39:58.544286Z [debug    ] Could not find tap.pipelinewise_logging.conf in /projects/.meltano/extractors/tap-rest-api-msdk/tap.pipelinewise_logging.conf, skipping.
2025-05-12T11:39:58.545931Z [debug    ] Invoking: ['/projects/.meltano/extractors/tap-rest-api-msdk/venv/bin/tap-rest-api-msdk', '--config', '/projects/.meltano/run/tap-rest-api-msdk/tap.058892ab-dba7-4bae-8dd4-14075e767e45.config.json', '--discover']
No schema found. Inferring schema from API call.
Error Connecting, message = {
  "detail": "resource not found",
  "error": "not found",
  "status": 404,
  "title": "resource not found",
  "type": "/webapp/view/api/errors.html#!/errors/not_found"
}
Traceback (most recent call last):
  File "/projects/.meltano/extractors/tap-rest-api-msdk/venv/bin/tap-rest-api-msdk", line 8, in <module>
    sys.exit(TapRestApiMsdk.cli())
  File "/projects/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.9/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
  File "/projects/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.9/site-packages/click/core.py", line 1081, in main
    with self.make_context(prog_name, args, **extra) as ctx:
  File "/projects/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.9/site-packages/click/core.py", line 949, in make_context
    self.parse_args(ctx, args)
  File "/projects/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.9/site-packages/click/core.py", line 1417, in parse_args
    value, args = param.handle_parse_result(ctx, opts, args)
  File "/projects/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.9/site-packages/click/core.py", line 2403, in handle_parse_result
    value = self.process_value(ctx, value)
  File "/projects/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.9/site-packages/click/core.py", line 2365, in process_value
    value = self.callback(ctx, self, value)
  File "/projects/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 539, in cb_discover
    tap.run_discovery()
  File "/projects/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 294, in run_discovery
    catalog_text = self.catalog_json_text
  File "/projects/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 314, in catalog_json_text
    return dump_json(self.catalog_dict, indent=2)
  File "/projects/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 305, in catalog_dict
    return t.cast(dict, self._singer_catalog.to_dict())
  File "/projects/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 325, in _singer_catalog
    for stream in self.streams.values()
  File "/projects/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 134, in streams
    for stream in self.load_streams():
  File "/projects/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 358, in load_streams
    for stream in self.discover_streams():
  File "/projects/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.9/site-packages/tap_rest_api_msdk/tap.py", line 494, in discover_streams
    schema = self.get_schema(
  File "/projects/.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.9/site-packages/tap_rest_api_msdk/tap.py", line 618, in get_schema
    raise ValueError(r.text)
ValueError: {
  "detail": "resource not found",
  "error": "not found",
  "status": 404,
  "title": "resource not found",
  "type": "/webapp/view/api/errors.html#!/errors/not_found"
}
2025-05-12T11:39:59.246640Z [debug    ] Deleted configuration at /projects/.meltano/run/tap-rest-api-msdk/tap.058892ab-dba7-4bae-8dd4-14075e767e45.config.json

I don't think there's much more you can do if the tap/SDK isn't logging anything useful out for you, other than to look at where this is happening in the code: https://github.com/Widen/tap-rest-api-msdk/blob/071b54032beecba75f6100b90281c1d88a47dfde/tap_rest_api_msdk/tap.py#L608-L619 Failing that, you could fork the tap repo and debug by adding some extra logging statements or with breakpoints.

Thanks for the pointer. That was very helpful. I missed some / between path and api_url...

Firstly, I suggest having a read on the Singer spec to get a general idea of how taps/targets communicate.

but the sqllight loader expects alle fields to be present

It would be the tap that is sending a

SCHEMA

message to the target, which is being used to create the tables in your SQLite db.

tap-rest-api-msdk

in particular does something generally referred to as "dynamic discovery", which is to say that (unless instructed otherwise) auto-generate a schema based on records returned directly by the stream API endpoint. The only configuration around this discovery behaviour appears to be num_inference_records, which you could try increasing from the default of

50

to expand the record sample size used to generate the schema (bear in mind this may have significant performance implications if you set it too high). You can also just set stream schema in config and bypass this dynamic discovery process entirely - that would be my preferred approach here. https://hub.meltano.com/extractors/tap-rest-api-msdk/#streams-setting

• schema: optional: A valid Singer schema or a path-like string that provides the path to a

.json

file that contains a valid Singer schema. If provided, the schema will not be inferred from the results of an api call.

Thank you very much for all the input. I'll have a look at it and see if I can make it work for my use case