Linden
02/12/2024, 6:23 AMpoetry run pytest
or instantiating the tap and running tap.run_connection_test()
. However, I'm pretty sure the url being generated by the RESTstream class is correct and works fine in postman. I am returning the full url by calling (streamer is instance of my custom stream class):
streamer.prepare_request(context=None, next_page_token=0).url
Would really appreciate some help on this!Edgar Ramírez (Arch.dev)
02/12/2024, 4:35 PMlaunch.json
example in the docs, but surely similar ways to set it up in any IDE.
You might also wanna check the query parameters.Linden
02/13/2024, 7:39 AMdef get_url_params(
self,
context: dict | None, # noqa: ARG002
next_page_token: Any | None, # noqa: ANN401
) -> dict[str, Any]:
"""Return a dictionary of values to be used in URL parameterization.
Args:
context: The stream context.
next_page_token: The next page index or value.
Returns:
A dictionary of URL query parameters.
"""
params: dict = {}
params['limit'] = self.config.get("limit", 100)
if next_page_token:
params["skip"] = next_page_token
if self.replication_key:
params["order"] = self.replication_key
return params
These params + the auth_token all get returned by calling streamer.prepare_request(context=None, next_page_token=0).url
as I would expect. I don't understand why the params seem correct but it still gives me the 400 error like they are not. Is there something else that gets added to the full request url during tap run?Edgar Ramírez (Arch.dev)
02/13/2024, 2:50 PMvalidate_response
and add a breakpoint
there to debug both the response and request:
class MyStream(RESTStream):
def validate_response(self, response):
breakpoint()
return super().validate_response(response)
and check response.request.url
, response.request.body
, response.request.headers
. Maybe also response.url
and response.history
.Linden
02/19/2024, 2:33 AMLinden
02/19/2024, 2:36 AM../../environvments/meltano/lib/python3.11/site-packages/singer_sdk/streams/core.py:409: in get_replication_key_signpost
return utc_now() if self.is_timestamp_replication_key else None
../../environvments/meltano/lib/python3.11/site-packages/singer_sdk/streams/core.py:218: in is_timestamp_replication_key
return is_datetime_type(type_dict)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
type_dict = None
def is_datetime_type(type_dict: dict) -> bool:
"""Return True if JSON Schema type definition is a 'date-time' type.
Also returns True if 'date-time' is nested within an 'anyOf' type Array.
"""
if not type_dict:
> raise EmptySchemaTypeError
E singer_sdk.helpers._typing.EmptySchemaTypeError: Could not detect type from empty type_dict. Did you forget to define a property in the stream schema?
Seems to be saying that my replication key is defined defined in my schema?
My schema definition inside my Stream class definition:
class EntriesStream(ContentfulStream):
"""Define Contenful Entries stream."""
name = "entries"
path = "/entries"
records_jsonpath = "$.items[*]"
primary_keys = ["sys.id"]
replication_key = "sys.updatedAt"
replication_key_param = "sys.updatedAt"
schema = th.PropertiesList(
th.Property("sys", th.ObjectType(
th.Property("id", th.StringType),
th.Property("updatedAt", th.DateTimeType),
)),
).to_dict()
Any ideas why this is happening?Edgar Ramírez (Arch.dev)
02/19/2024, 4:58 PMLinden
02/19/2024, 11:22 PM