Hey folks, currently developing my first custom ta...
# singer-tap-development
l
Hey folks, currently developing my first custom tap for contentful (existing ones not fit for purpose). I'm stuck on getting a 400 Bad Request error when either running
poetry run pytest
or instantiating the tap and running
tap.run_connection_test()
. However, I'm pretty sure the url being generated by the RESTstream class is correct and works fine in postman. I am returning the full url by calling (streamer is instance of my custom stream class):
Copy code
streamer.prepare_request(context=None, next_page_token=0).url
Would really appreciate some help on this!
1
e
Hi @Linden! Have you tried debugging with your IDE? There's a VSCode
launch.json
example in the docs, but surely similar ways to set it up in any IDE. You might also wanna check the query parameters.
l
Thanks for the reply @Edgar Ramírez (Arch.dev). Running the VSCode debugger I get the same issue. AFAIK the query params are correct, based on what comes out of that command I posted above, and also based on how I understand the get_url_params method should be set up:
Copy code
def get_url_params(
        self,
        context: dict | None,  # noqa: ARG002
        next_page_token: Any | None,  # noqa: ANN401
    ) -> dict[str, Any]:
        """Return a dictionary of values to be used in URL parameterization.

        Args:
            context: The stream context.
            next_page_token: The next page index or value.

        Returns:
            A dictionary of URL query parameters.
        """
        params: dict = {}
        params['limit'] = self.config.get("limit", 100)
        if next_page_token:
            params["skip"] = next_page_token
        if self.replication_key:
            params["order"] = self.replication_key
        return params
These params + the auth_token all get returned by calling
streamer.prepare_request(context=None, next_page_token=0).url
as I would expect. I don't understand why the params seem correct but it still gives me the 400 error like they are not. Is there something else that gets added to the full request url during tap run?
e
in cases like this I'd override
validate_response
and add a
breakpoint
there to debug both the response and request:
Copy code
class MyStream(RESTStream):
    def validate_response(self, response):
        breakpoint()
        return super().validate_response(response)
and check
response.request.url
,
response.request.body
,
response.request.headers
. Maybe also
response.url
and
response.history
.
l
Thanks @Edgar Ramírez (Arch.dev), I was able to work it out!
Have run into another error
Copy code
../../environvments/meltano/lib/python3.11/site-packages/singer_sdk/streams/core.py:409: in get_replication_key_signpost
    return utc_now() if self.is_timestamp_replication_key else None
../../environvments/meltano/lib/python3.11/site-packages/singer_sdk/streams/core.py:218: in is_timestamp_replication_key
    return is_datetime_type(type_dict)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

type_dict = None

    def is_datetime_type(type_dict: dict) -> bool:
        """Return True if JSON Schema type definition is a 'date-time' type.
    
        Also returns True if 'date-time' is nested within an 'anyOf' type Array.
        """
        if not type_dict:
>           raise EmptySchemaTypeError
E           singer_sdk.helpers._typing.EmptySchemaTypeError: Could not detect type from empty type_dict. Did you forget to define a property in the stream schema?
Seems to be saying that my replication key is defined defined in my schema? My schema definition inside my Stream class definition:
Copy code
class EntriesStream(ContentfulStream):
    """Define Contenful Entries stream."""

    name = "entries"
    path = "/entries"
    records_jsonpath = "$.items[*]"
    primary_keys = ["sys.id"]
    replication_key = "sys.updatedAt"
    replication_key_param = "sys.updatedAt"
    schema = th.PropertiesList(
        th.Property("sys", th.ObjectType(
            th.Property("id", th.StringType),
            th.Property("updatedAt", th.DateTimeType),
        )),
    ).to_dict()
Any ideas why this is happening?
e
yeah, nested replication keys are not supported at the moment unfortunately: https://github.com/meltano/sdk/issues/1198. There's a workaround in that issue, so you could give that a try!
l
Yes thanks that worked! Got it all working now, appreciate your help @Edgar Ramírez (Arch.dev)
🙌 1