They team, Im having a hell of a time trying to fi...
# getting-started
j
They team, Im having a hell of a time trying to figure out how to pass config values to my custom extractor and I can figure out what the documentation is telling me to do. Is there a simplified, step by step, document anywhere that just says "do x, then y"?
1
r
Have you looked at https://docs.meltano.com/tutorials/custom-extractor#2-edit-your-existing-meltanoyml-file? For a custom plugin, you need to define
settings
for the plugin in your
meltano.yml
and the corresponding
config
(either also in the
meltano.yml
or via environment variables - see https://docs.meltano.com/guide/configuration/#configuration-layers)
👍 1
j
I have but I don't know how to call them in my custom connection. Like how do I pass them to a stream or a client? I can define the yml, that's not the hard part
r
I assume you are using the SDK then? In the client/stream classes, you have access to
self.config
in all instance methods, which is the configuration passed to the plugin.
👍 1
j
Ah ok. Let me give that a shot and I'll get back to you. Thank you!
So here's my tap class
Copy code
class Tapjsonplaceholder(Tap):
    """json placeholder tap class."""

    name = "tap-jsonplaceholder"

    config_jsonschema = th.PropertiesList(
        th.Property(
            "model",
            th.StringType,
            description="Model",
        ),
        th.Property(
            "api_key",
            th.StringType,
            description="Model",
        ),
        th.Property(
            "content_type",
            th.StringType,
            description="JSON Content Type",
        ),
        th.Property(
            "base_url",
            th.StringType,
            description="Base Url",
        )
    ).to_dict()
then do I add it my stream like this?
Copy code
class TableDataStream(jsonplaceholderStream):

    def __int__(
            self,
            tap: Any,
            name: str,
            model: str,
            content_type: str,
            base_url: str,
            api_key: str,
            schema: Optional[dict] = None,
        ) -> None:
            """Class initialization.

            Args:
                tap: see tap.py
                name: see tap.py
                model: see tap.py
                content_type: see tap.py
                base_url: see tap.py
                api_key: see tap.py
            """
            super().__init__(tap=tap, name=tap.name, schema=schema)
            self.name = name
            self.model = model
            self.content_type = content_type
            self.base_url = base_url
            self.api_key = api_key

    primary_keys = ["PayeeID_"]
    path = '/inputforms/0/data'
    name = "tableData"
    schema = th.PropertiesList(
        th.Property("PayeeID_", th.StringType),
        th.Property("CompPlanID", th.StringType),
        th.Property("TeamID", th.StringType),
        th.Property("Periods", th.StringType),
        th.Property("RunningTotalCredit", th.NumberType),
        th.Property("Quota", th.NumberType),
        th.Property("MonthlyAttainmentIncrease", th.NumberType),
        th.Property("StatusID", th.StringType),
        th.Property("ComponentWeighting", th.NumberType),
        th.Property("RampWeight", th.NumberType),
        th.Property("AttributeID", th.StringType),
    ).to_dict()
r
Can you show
jsonplaceholderStream
? I think you want to be configuring an Authenticator class: https://github.com/Matatika/tap-auth0/blob/3daf0da096520fb0d3ab2712346e09f64059721e/tap_auth0/client.py#L21-L24 https://github.com/Matatika/tap-auth0/blob/3daf0da096520fb0d3ab2712346e09f64059721e/tap_auth0/auth.py Generally, you shouldn't need to redefine
__init__
except for special cases.
At least, that looks like what your config points towards (
api_key
,
content_type
,
base_url
). You could probably reference more of the
tap-auth0
client class (
Auth0Stream
) as an example here, since that has a configurable
domain
similar to how I imagine
base_url
will work in your case. https://github.com/Matatika/tap-auth0/blob/3daf0da096520fb0d3ab2712346e09f64059721e/tap_auth0/client.py
j
Ah ok I see, thank you
This is the
tapjsonplaceholderStream
Copy code
class jsonplaceholderStream(RESTStream):
    """jsonplaceholder stream class."""

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

    @property
    def url_base(self) -> str:
        """Return the API URL root, configurable via tap settings."""
        # TODO: hardcode a value here, or retrieve it from self.config
        return self.config.get("base_url")

    records_jsonpath = "$[*]"  # Or override `parse_response`.

    # Set this value or override `get_new_paginator`.
    next_page_token_jsonpath = "$.next_page"  # noqa: S105

    @property
    def authenticator(self):
        return SimpleAuthenticator(
            stream=self,
            auth_headers={
                "Authorization": f"Bearer {self.config.get('api_key')}",
            },
        )

    @property
    def http_headers(self) -> dict:
        """Return the http headers needed.

        Returns:
            A dictionary of HTTP headers.
        """
        headers = {}
        headers["model"] = self.congin.get("model")
        headers["Content-Type"] = self.config.get("content_type")
        headers["Authorization"] = f"Bearer {self.config.get('api_key')}"
        if "user_agent" in self.config:
            headers["User-Agent"] = self.config.get("user_agent"),
        return headers

    def get_new_paginator(self) -> BaseAPIPaginator:
        """Create a new pagination helper instance.

        If the source API can make use of the `next_page_token_jsonpath`
        attribute, or it contains a `X-Next-Page` header in the response
        then you can remove this method.

        If you need custom pagination that uses page numbers, "next" links, or
        other approaches, please read the guide: <https://sdk.meltano.com/en/v0.25.0/guides/pagination-classes.html>.

        Returns:
            A pagination helper instance.
        """
        return super().get_new_paginator()

    def get_url_params(
        self,
        context: dict | None,  # noqa: ARG002
        next_page_token: Any | None,  # noqa: ANN401
    ) -> dict[str, Any]:
        """Return a dictionary of values to be used in URL parameterization.

        Args:
            context: The stream context.
            next_page_token: The next page index or value.

        Returns:
            A dictionary of URL query parameters.
        """
        params: dict = {}
        if next_page_token:
            params["page"] = next_page_token
        if self.replication_key:
            params["sort"] = "asc"
            params["order_by"] = self.replication_key
        return params


    def parse_response(self, response: requests.Response) -> Iterable[dict]:
        """Parse the response and return an iterator of result records.

        Args:
            response: The HTTP ``requests.Response`` object.

        Yields:
            Each record from the source.
        """
        # TODO: Parse response body and return a set of records.
        keys = [column['name'] for column in response.json()['columnDefinitions']]
        combined_data = []

        for data_row in response.json()['data']:
            if len(keys) != len(data_row):
                raise ValueError('Mismatch between keys and values length')
            combined_data.append(dict(zip(keys, data_row)))

        yield from extract_jsonpath('$[*]', input=combined_data)
r
Looks OK to me - I think you just need to get rid of the
__init__
override in
TableDataStream
, and then make sure the stream is referenced in the
discover_streams
method of you tap class (
tap.py
): https://github.com/Matatika/tap-auth0/blob/3daf0da096520fb0d3ab2712346e09f64059721e/tap_auth0/tap.py#L53-L54
j
Ok let me try that. I really appreciate the help!
Hey Ruben, Thanks for the help, it all worked out!
r
No problem, glad you got it working! 😄