Hi Everyone I am trying to include stats fields f...
# singer-taps
s
Hi Everyone I am trying to include stats fields for tickets data on tap-freshdesk and am getting error as below [ it works well without stats embedded option and was able to get other entities tickets,roles,skills,agents etc data ] 'message': "singer_sdk.exceptions.FatalAPIError: 400 Client Error: Bad Request for path: /api/v2/tickets. Error via function response_error_message : Error 1: Message - It should be a valid date in the 'combined date and time ISO8601' format, Field - updated_since." Below is the current client.py def get_url_params( self, context: dict | None, next_page_token: Any | None, ) -> dict[str, Any]: """Return a dictionary of values to be used in URL parameterization. Args: context: The stream context. next_page_token: The next page index or value. Returns: A dictionary of URL query parameters. """ params: dict = {} embeds = self.config.get("embeds") if embeds: embed_fields = embeds.get(self.name, []) required_fields = ["stats"] for field in required_fields: if field not in embed_fields: embed_fields.append(field) # Check if embed_fields is not empty if embed_fields: params["include"] = ",".join(embed_fields) return params can you help here - Thank you
e
Did you define a
start_date
for the extractor?
a
Thanks, looks like I should make
start_date
required in the tap schema.
👍 1
s
I defined it meltano.yml file
Copy code
config:
  start_date: 2024-08-10T00:00:01Z
after troubleshooting tap full url looks like https://vvv.freshdesk.com/api/v2/tickets?updated_since=2024-08-10+00:00:01+00:00&per_page=100&page=1&order_type=asc&order_by=updated_at Causing error singer_sdk.exceptions.FatalAPIError: 400 Client Error: Bad Request for path: /api/v2/tickets. Error via func tion response_error_message : Error 1: Message - It should be a valid date in the 'combined date and time ISO8601' format, Field - updated_since. any idea about this !
a
Just leave it as a iso date
Copy code
config:
  start_date: 2024-08-10
s
same error with start_date: 2024-08-10
I have client.py as below
Copy code
"""REST client handling, including freshdeskStream base class."""

from __future__ import annotations
import time

from pathlib import Path
from typing import Any, Callable, Iterable, TYPE_CHECKING, Generator

import requests
from http import HTTPStatus
from urllib.parse import urlparse
from singer_sdk.authenticators import BasicAuthenticator
from singer_sdk.helpers.jsonpath import extract_jsonpath
from singer_sdk.streams import RESTStream
from singer_sdk.pagination import BasePageNumberPaginator, SinglePagePaginator

if TYPE_CHECKING:
    from requests import Response

_Auth = Callable[[requests.PreparedRequest], requests.PreparedRequest]
SCHEMAS_DIR = Path(__file__).parent / Path("./schemas")

class FreshdeskStream(RESTStream):
    """freshdesk stream class."""

    name: str
    records_jsonpath = "$.[*]"  # Or override `parse_response`.
    primary_keys = ["id"]
    @property
    def backoff_max_tries(self) -> int:
        return 10

    @property
    def path(self) -> str:
        return f"/{self.name}"

    @property
    def schema_filepath(self) -> Path | None:
        return SCHEMAS_DIR / f"{self.name}.json"

    # OR use a dynamic url_base:
    @property
    def url_base(self) -> str:
        domain = self.config["domain"]
        return f"https://{domain}.<http://freshdesk.com/api/v2|freshdesk.com/api/v2>"

    @property
    def authenticator(self) -> BasicAuthenticator:
        return BasicAuthenticator.create_for_stream(
            self,
            username=self.config.get("api_key", ""),
            password="",
        )

    @property
    def http_headers(self) -> dict:

        headers = {}
        if "user_agent" in self.config:
            headers["User-Agent"] = self.config.get("user_agent")
        return headers

    def get_next_page_token(
        self,
        response: requests.Response,
        previous_token: Any | None,
    ) -> Any | None:

        if self.next_page_token_jsonpath:
            all_matches = extract_jsonpath(
                self.next_page_token_jsonpath, response.json()
            )
            first_match = next(iter(all_matches), None)
            next_page_token = first_match
        else:
            next_page_token = response.headers.get("X-Next-Page", None)

        return next_page_token

    def get_url_params(
        self,
        context: dict | None,
        next_page_token: Any | None,
    ) -> dict[str, Any]:

        params: dict = {}
        embeds = self.config.get("embeds")
        if embeds:
            embed_fields = embeds.get(self.name, [])
            if embed_fields:  # i.e. 'stats,company,sla_policy'
                params["include"] = ",".join(embed_fields)
        return params

    def parse_response(self, response: requests.Response) -> Iterable[dict]:

        yield from extract_jsonpath(self.records_jsonpath, input=response.json())

    def get_new_paginator(self) -> SinglePagePaginator:
        return SinglePagePaginator()

    def backoff_wait_generator(self) -> Generator[float, None, None]:
        return self.backoff_runtime(value=self._wait_for)

    @staticmethod
    def _wait_for(exception) -> int:

        return int(exception.response.headers["Retry-After"])

    def backoff_jitter(self, value: float) -> float:
        return value

    # Handling error, overriding this method over RESTStream's Class
    def response_error_message(self, response: requests.Response) -> str:

        full_path = urlparse(response.url).path or self.path
        error_type = (
            "Client"
            if HTTPStatus.BAD_REQUEST
            <= response.status_code
            < HTTPStatus.INTERNAL_SERVER_ERROR
            else "Server"
        )

        error_details = []
        if response.status_code >= 400:
            print(f"Error Response: {response.status_code} {response.reason}")
            try:
                error_data = response.json()
                errors = error_data.get("errors")
                for index, error in enumerate(errors):
                    message = error.get("message", "Unknown")
                    field = error.get("field", "Unknown")
                    error_details.append(
                        f"Error {index + 1}: Message - {message}, Field - {field}"
                    )
            except requests.exceptions.JSONDecodeError:
                return "Error: Unable to parse JSON error response"

        return (
            f"{response.status_code} {error_type} Error: "
            f"{response.reason} for path: {full_path}. "
            f"Error via function response_error_message : {'. '.join(error_details)}."
        )


class FreshdeskPaginator(BasePageNumberPaginator):

    def has_more(self, response: Response) -> bool:
        return len(response.json()) != 0 and self.current_value < 300


class PagedFreshdeskStream(FreshdeskStream):

    def get_url_params(
        self,
        context: dict | None,
        next_page_token: Any | None,
    ) -> dict[str, Any]:

        context = context or {}
        params = super().get_url_params(context, next_page_token)
        params["per_page"] = 100
        if next_page_token:
            params["page"] = next_page_token
        if "updated_since" not in context:
            params["updated_since"] = self.get_starting_timestamp(context)
        return params

    def get_new_paginator(self) -> BasePageNumberPaginator:
        return FreshdeskPaginator(start_value=1)

class PagedFreshdeskTicketsStream(FreshdeskStream):
    def get_url_params(
        self,
        context: dict | None,
        next_page_token: Any | None,
    ) -> dict[str, Any]:
        context = context or {}
        params = super().get_url_params(context, next_page_token)
        params["per_page"] = 30
        if next_page_token:
            params["page"] = next_page_token
        if "updated_since" not in context:
            params["updated_since"] = self.get_starting_timestamp(context)
        return params

    def get_new_paginator(self) -> BasePageNumberPaginator:
        return FreshdeskPaginator(start_value=1)
e
If you're using
meltano run
, can you use the
--full-refresh
flag too?
a
And please share the full url now too?
👍 1
s
Below are the details for reference start_date : 2024-08-15 select : tickets.* URL : https://aaa.freshdesk.com/api/v2/tickets?updated_since=2024-08-15+00:00:00+00:00&amp;per_page=100&amp;page=1&amp;order_type=asc&amp;order_by=updated_at Error: singer_sdk.exceptions.FatalAPIError: 400 Client Error: Bad Request for path: /api/v2/tickets. Error via function response_error_message : Error 1: Message - It should be a valid date in the 'combined date and time ISO8601' format, Field - updated_since.
a
It looks like the full timestamp is still going to the tap, which explains why the error has not changed. Can you please share full
meltano.yml
? @Edgar Ramírez (Arch.dev) is it possible this config value could be cached somewhere?
s
version: 1 send_anonymous_usage_stats: true project_id: tap-freshdesk default_environment: test environments: - name: test plugins: extractors: - name: tap-freshdesk namespace: tap_freshdesk pip_url: -e . capabilities: - state - catalog - discover - about - stream-maps settings: - name: username - name: password - name: start_date - name: row_count - name: batch_size config: start_date: 2024-08-15 domain: domain api_key: key row_count: 1000 batch_size: 1000 select: - business_hours.* - tickets.* loaders: - name: target-jsonl variant: andyh1203 pip_url: target-jsonl elt: buffer_size: 52428800
a
Are you suppling any environmental variables that might override
TAP_FRESHDESK_START_DATE
?
Can you try
meltano config tap-freshdesk list
To see what the source of the start date is?
s
image.png
it's supplied from yml
a
Can you change the date to
2024-08-10
in meltano.yml and see if the URL changes? Or is the timestamp value cached from somewhere?
Have you modified the tap at all? Not sure why username and password are declared as settings. Without seeing your full code not sure I can help here any further, I am using my repo as a reference but your code could be edited. https://github.com/acarter24/tap-freshdesk/blob/main/tap_freshdesk/tap.py
s
Thank you @Andy Carter seems to be python version compatibility issues some how I was able to run in py 3.9
👍 1