Thank you @visch @aaronsteers @pat_nadolny @edgar_ramirez_mondragon for all the helpful comments in the office hours call. I don't know if the chat gets saved somewhere, but this time I quickly copied some of the useful links and jotted them down here: As a developer of a tap, I need a quick start guide that takes me through the first few hours of developing and debugging my tap. Ideally I'd follow a simple 1,2,3 guide, improve on: https://sdk.meltano.com/en/latest/dev_guide.html#using-an-existing-library https://gitlab.com/meltano/sdk/-/issues/329 1. Create a new tap using SDK cookie cutter Notes from ‘Feb tapfest 2022’ - this was easy enough, I got past the pipx virtual environment issues more successfully this time. 2. Work through TODO’s.  Running the tap with poetry until no obvious failures

poetry run tap-shopify --about

3. Work through TODO’s.  Running the tap with meltano switching between Singer stream and my debug’s.   https://gitlab.com/meltano/meltano/-/issues/3281 https://gitlab.com/meltano/sdk/-/issues/170 Debug Singer json stream

meltano invoke tap-shopify 2>/dev/null

Add my debugs with with self.logging.info - these will go to stderr by default

<http://self.logging.info|self.logging.info>("hello world")

meltano invoke tap-shopify 2>/dev/null

4. Debugging the https requests in a REST tap is useful. URI is available by _LOG_REQUEST_METRIC_URLS on the Stream.

class PubgStream(RESTStream):
     """pubg stream class."""
     _LOG_REQUEST_METRIC_URLS=True

Notes from ‘Feb tapfest 2022’ - using _LOG_REQUEST_METRIC_URLS doesn’t log the actual http requests.  It happened that I overrode url_base() and had set a default configuration value for ‘admin_url’.  Consequently my base uri was not what I thought it should be:

@property
    def url_base(self) -> str:
        """Return the API URL root, configurable via tap settings."""
        url_base = self.config.get("admin_url") or '<https://%s.myshopify.com/admin>' % self.config.get("store")
        return url_base

5. The core SDK tests are really useful, but as they call real endpoints, these fail on first test iterations. Notes from ‘Feb tapfest 2022’ - I found it was not apparent why, or what development should be targeted next.  In our case the credentials for the API won’t be stored in the repository.  So the expectation is that we’ll mock the responses to get these tests to pass. Requests caching feature proposal: https://gitlab.com/meltano/sdk/-/issues/237 Stephen Bailey: https://gitlab.com/meltano/sdk/-/merge_requests/197 6. Stream types.  In the first few iterations we need to build Streams with types. Generating types with genson looks like a good idea: https://pypi.org/project/genson/ There is an issue to auto-document typing helper classes: https://gitlab.com/meltano/sdk/-/issues/332 Auto-detecting schema: https://gitlab.com/meltano/sdk/-/issues/174 Notes from ‘Feb tapfest 2022’ - Felt like the guides ran out at this point. Creating a selection of custom types in the SDK could be useful, but I’ve found that the CurrencyAmount did not turn out correctly in postgres.  To be investigated before creating an issue or MR.

class IPv4Type(JSONTypeHelper):
@classproperty
def type_dict(cls) -> dict:
return {
"type": ["string"],
"format": ["ipv4"],
}

class CurrencyAmount(JSONTypeHelper):
@classproperty
def type_dict(cls) -> dict:
return {
"type": ["string"],
"pattern": ["^(0|([1-9]+[0-9]*))(\.[0-9]{1,2})?$"],
"minLength": 1,
}