In my custom tap, how can I access the catalog passed via

--catalog catalog.json

so I don’t have to rediscover my schema? Or is that catalog.json only sent to the target

Does your tap perform dynamic discovery, or is the catalog mostly static?

Yeah dynamic discovery I’ve got a catalog.json that I’d like to reuse on subsequent runs. Motivation: ensure that everything looks like we expected it to when we locked in the catalog.json

I think the docstring is a bit misleading. The catalog does specify the schema, but the catalog's schema is used in a different place: https://github.com/meltano/sdk/blob/e79d755ff8f16264f3fd45787dd3ac8eeba31fd6/singer_sdk/mapper.py#L678-L693

Wait that’s in mapper? I couldn’t for the life of me figure out how to get this simple behaviour out of the box while developing a new tap: 1. Run

--discover > catalog.json

2. Run

--catalog catalog.json

Took a while to wrap my head around all the existing streams logic. Ended up overriding

Tap.discover_streams

and

Stream.apply_catalog

to get the behaviour I was looking for

Yeah, mappers can fan out a single stream into multiple ones so they get passed the input schema. By default the mapper uses the schema straight from the input catalog.

Ok that makes sense, not using mappers in this scenario here. What’s the best way to implement the behaviour I mentioned above? So much of Meltano is a black box to me, like how

tap.properties.json

is generated etc

I think the default

SQLTap

implementation does what you want: https://github.com/meltano/sdk/blob/0eb478c0416235db81e2aa15d7238cf252b84ec6/singer_sdk/tap_base.py#L665-L699

That’s what my current version is based on! Took a while to get through the tangle of similar-looking functions calling each other. Really appreciate you sharing. If that’s the best we’ve got as an example, then I do think we’re lacking a bit. What makes Meltano so promising to me is its extensibility, but a cookbook of examples or basic API docs would go a long way here. Maybe I can contribute

We do have https://sdk.meltano.com/en/v0.36.1/classes/singer_sdk.SQLTap.html, but I we're certainly open to improvements in docs structure, content and discoverability 😁. What in particular do you think is missing?

I like the page of recipes, but it’s very basic and some of it outdated. Even a link to the SQL Tap there would help discover “how to write your own mapper”