Jared Garvin-Joseph
02/08/2024, 5:59 AMReuben (Matatika)
02/08/2024, 8:11 AMsettings
for the plugin in your meltano.yml
and the corresponding config
(either also in the meltano.yml
or via environment variables - see https://docs.meltano.com/guide/configuration/#configuration-layers)Jared Garvin-Joseph
02/08/2024, 4:22 PMReuben (Matatika)
02/08/2024, 4:35 PMself.config
in all instance methods, which is the configuration passed to the plugin.Jared Garvin-Joseph
02/08/2024, 5:53 PMJared Garvin-Joseph
02/08/2024, 5:54 PMclass Tapjsonplaceholder(Tap):
"""json placeholder tap class."""
name = "tap-jsonplaceholder"
config_jsonschema = th.PropertiesList(
th.Property(
"model",
th.StringType,
description="Model",
),
th.Property(
"api_key",
th.StringType,
description="Model",
),
th.Property(
"content_type",
th.StringType,
description="JSON Content Type",
),
th.Property(
"base_url",
th.StringType,
description="Base Url",
)
).to_dict()
Jared Garvin-Joseph
02/08/2024, 5:55 PMclass TableDataStream(jsonplaceholderStream):
def __int__(
self,
tap: Any,
name: str,
model: str,
content_type: str,
base_url: str,
api_key: str,
schema: Optional[dict] = None,
) -> None:
"""Class initialization.
Args:
tap: see tap.py
name: see tap.py
model: see tap.py
content_type: see tap.py
base_url: see tap.py
api_key: see tap.py
"""
super().__init__(tap=tap, name=tap.name, schema=schema)
self.name = name
self.model = model
self.content_type = content_type
self.base_url = base_url
self.api_key = api_key
primary_keys = ["PayeeID_"]
path = '/inputforms/0/data'
name = "tableData"
schema = th.PropertiesList(
th.Property("PayeeID_", th.StringType),
th.Property("CompPlanID", th.StringType),
th.Property("TeamID", th.StringType),
th.Property("Periods", th.StringType),
th.Property("RunningTotalCredit", th.NumberType),
th.Property("Quota", th.NumberType),
th.Property("MonthlyAttainmentIncrease", th.NumberType),
th.Property("StatusID", th.StringType),
th.Property("ComponentWeighting", th.NumberType),
th.Property("RampWeight", th.NumberType),
th.Property("AttributeID", th.StringType),
).to_dict()
Reuben (Matatika)
02/08/2024, 6:08 PMjsonplaceholderStream
? I think you want to be configuring an Authenticator class:
https://github.com/Matatika/tap-auth0/blob/3daf0da096520fb0d3ab2712346e09f64059721e/tap_auth0/client.py#L21-L24
https://github.com/Matatika/tap-auth0/blob/3daf0da096520fb0d3ab2712346e09f64059721e/tap_auth0/auth.py
Generally, you shouldn't need to redefine __init__
except for special cases.Reuben (Matatika)
02/08/2024, 6:11 PMapi_key
, content_type
, base_url
). You could probably reference more of the tap-auth0
client class (Auth0Stream
) as an example here, since that has a configurable domain
similar to how I imagine base_url
will work in your case.
https://github.com/Matatika/tap-auth0/blob/3daf0da096520fb0d3ab2712346e09f64059721e/tap_auth0/client.pyJared Garvin-Joseph
02/08/2024, 6:17 PMJared Garvin-Joseph
02/08/2024, 6:18 PMtapjsonplaceholderStream
Jared Garvin-Joseph
02/08/2024, 6:18 PMclass jsonplaceholderStream(RESTStream):
"""jsonplaceholder stream class."""
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
@property
def url_base(self) -> str:
"""Return the API URL root, configurable via tap settings."""
# TODO: hardcode a value here, or retrieve it from self.config
return self.config.get("base_url")
records_jsonpath = "$[*]" # Or override `parse_response`.
# Set this value or override `get_new_paginator`.
next_page_token_jsonpath = "$.next_page" # noqa: S105
@property
def authenticator(self):
return SimpleAuthenticator(
stream=self,
auth_headers={
"Authorization": f"Bearer {self.config.get('api_key')}",
},
)
@property
def http_headers(self) -> dict:
"""Return the http headers needed.
Returns:
A dictionary of HTTP headers.
"""
headers = {}
headers["model"] = self.congin.get("model")
headers["Content-Type"] = self.config.get("content_type")
headers["Authorization"] = f"Bearer {self.config.get('api_key')}"
if "user_agent" in self.config:
headers["User-Agent"] = self.config.get("user_agent"),
return headers
def get_new_paginator(self) -> BaseAPIPaginator:
"""Create a new pagination helper instance.
If the source API can make use of the `next_page_token_jsonpath`
attribute, or it contains a `X-Next-Page` header in the response
then you can remove this method.
If you need custom pagination that uses page numbers, "next" links, or
other approaches, please read the guide: <https://sdk.meltano.com/en/v0.25.0/guides/pagination-classes.html>.
Returns:
A pagination helper instance.
"""
return super().get_new_paginator()
def get_url_params(
self,
context: dict | None, # noqa: ARG002
next_page_token: Any | None, # noqa: ANN401
) -> dict[str, Any]:
"""Return a dictionary of values to be used in URL parameterization.
Args:
context: The stream context.
next_page_token: The next page index or value.
Returns:
A dictionary of URL query parameters.
"""
params: dict = {}
if next_page_token:
params["page"] = next_page_token
if self.replication_key:
params["sort"] = "asc"
params["order_by"] = self.replication_key
return params
def parse_response(self, response: requests.Response) -> Iterable[dict]:
"""Parse the response and return an iterator of result records.
Args:
response: The HTTP ``requests.Response`` object.
Yields:
Each record from the source.
"""
# TODO: Parse response body and return a set of records.
keys = [column['name'] for column in response.json()['columnDefinitions']]
combined_data = []
for data_row in response.json()['data']:
if len(keys) != len(data_row):
raise ValueError('Mismatch between keys and values length')
combined_data.append(dict(zip(keys, data_row)))
yield from extract_jsonpath('$[*]', input=combined_data)
Reuben (Matatika)
02/08/2024, 6:23 PM__init__
override in TableDataStream
, and then make sure the stream is referenced in the discover_streams
method of you tap class (tap.py
):
https://github.com/Matatika/tap-auth0/blob/3daf0da096520fb0d3ab2712346e09f64059721e/tap_auth0/tap.py#L53-L54Jared Garvin-Joseph
02/08/2024, 6:29 PMJared Garvin-Joseph
02/08/2024, 11:40 PMReuben (Matatika)
02/08/2024, 11:45 PM