Linen
Join Slack
Hi everyone, I'm encountering an issue with `tap-freshdesk` while trying to extract data. Specifical...
# troubleshooting
p
Hi everyone, I'm encountering an issue with 
tap-freshdesk
while trying to extract data. Specifically, I'm receiving a 'FatalAPIError' with a 400 status code. Has anyone encountered a similar issue before or have any insights on how to resolve it? Issue Description: Unhandled Error Encountered While Syncing 'tickets_abridged' in Meltano An unhandled error is occurring during the synchronization process of the 'tickets_abridged' data stream while transferring data from Freshdesk to Snowflake using Meltano's tap-freshdesk and target-snowflake connectors. The issue arises specifically when executing the sync operation, resulting in a 
FatalAPIError
being raised with the associated error message. Error Message:
Copy code
raise FatalAPIError(msg)
[info] singer_sdk.exceptions.FatalAPIError: 400 Client Error: Bad Request for path: /api/v2/tickets
Detailed Description: Upon initial execution, the data transfer process successfully moves a significant portion of the data for all other schemas and failed for tickets_abridged after moving few thousand records leading to incomplete data synchronization. Even after re-running the operation, only a fraction of the remaining data for tickets_abridged is getting transferred before encountering the same error again. Any guidance is much appreciated. Issue link: tap-freshdesk issue @Andy Carter
a
Hi, I don't have access to snowflake to test this particular configuration. Can you run the rap via 
meltano invoke tap-freshdesk
and see if the same error occurs? Do you get any output at all from other streams before the 400 error? Can you check your credentials / API are correct using the freshdesk api directly via postman or similar?
Please share your 
meltano.yml
with credentials removed too if you can
p
Credentials/API is correct since it has dumped data for all other schemas including some of 
tickets_abridged
but failed after dumping few records for tickets_abridged. By far what we've dubugged we think it's a pagination issue where it's not rightly interpreting the last page hence returning the 
FatalAPIError: 400 Client Error: Bad Request for path: /api/v2/tickets
a
Thanks. So you are getting at least page of records from tickets_abridged but then the error occurs. Honestly I haven't seen this error before. Do you have any idea of the total number of tickets you have in freshdesk?
Are you definitely using the latest version of the tap too? What pipurl are you using? have you done a recent reinstall of the tap using 
--clean
?
I can't replicate running the tap locally but then we only have around 4-500 tickets
Both the abridged and detail stream use the 
api/v2/tickets
path so it could be either
The stream sync can take a long time, so it is possible that a ticket got deleted in the process? The tap using 
abridged
stream to find the tickets it needs to read 
api/v2/tickets
, then attempts to read each one specifically at 
api/v2/tickets/12345
to get more detail. If ticket 12345 has been deleted between the abridged run then the detail run might fail.
Hmm I don't think it's that. If the ticket ID cannot be found at the detail level then the error returned is 
404 Client Error: Not Found for path: /api/v2/tickets/1000
Can you replicate the error and share the traceback here?
p
We've around 30k+ tickets. We've just set it up for the first time so it'd be the latest version only.
DMing you the error traceback, please check. @Andy Carter
a
What meltano command are you running?
And please share the section of 
meltano.yml
for the freshdesk tap without api key
Can you please test with a basic target like 
target-jsonl
or just a simple 
meltano invoke tap-freshdesk
to see if it's a target issue
I only have experience running the tap with postgres
Also you could test a different variant and see if that works for you? https://hub.meltano.com/extractors/tap-freshdesk--singer-io/
m
I think we are able to identify the issue. Currently, there's no limit on number of pages and as soon as you pass the value 
page=301
, it throws 400 error. We looked at the code and it doesn't specify when it stop (if page 300 still has non empty data). Though we are a bit confused why would API return so much data even when we are passing today's date in 
updated_since
a
That makes sense, the API will only list up to 300 pages of tickets by design: https://developers.freshdesk.com/api/#list_all_tickets I will see if I can make a fix for this
m
We have made some other changes also. Will gladly send a PR once we have tested it 🙂
👍 1
We noticed one more thing. Ideally the schema of response for any endpoint should remain the same for all org and customisation like custom fields should be added to 
custom_fields
key but noticed in our case there are keys that are coming through at top level rather than 
custom_fields
in some of the streams (e.g. contacts, tickets_abridged). How do you think we can handle this? We are getting warning like this:
Copy code
Properties ('csat_rating', 'preferred_source', 'other_companies', 'unique_external_id', 'first_name', 'last_name', 'visitor_id', 'org_contact_id', 'other_phone_numbers') were present in the 'contacts' stream but not found in catalog schema.
a
Are you doing any unpacking of json objects in your snowflake target?
We do not use any custom fields in our deployment so not sure how to incorporate this. But the tap should not be unpacking the 
custom_fields
object into top-level keys. Could you test with 
target-jsonl
to see if this is snowflake-specific behaviour please?
m
No I meant this is not an implementation issue but more how freshdesk is sending extra fields at top level. Not very clear how it gets implemented but will check internally to understand better
a
That's why I'm asking you to try a different tap to isolate the behaviour to the API. Are you enabling flattening on your target? https://hub.meltano.com/loaders/target-snowflake#flattening_enabled-setting
m
I don't think it's an issue with the tap. I directly called the api and that's how the response is. Will reach out to freshdesk support team to ask them about this
👍 1
p
Hi @Andy Carter, I've raised the PR for the issues we were facing, link : https://github.com/acarter24/tap-freshdesk/pull/15. Please review and let me know if you need any clarification on this. We've ran and tested it end to end for our system and it's working as expected.
a
Thanks, will give it a test. Would you consider being a maintainer of the project?
p
Hi @Andy Carter, I've implemented the changes as suggested. Please review the PR again. Also, regarding the role of maintainer for the project, since this would be my first time contributing to an open-source project, I believe I can take on the responsibility alongside you as a moderator.
a
@Prashant Vikram thanks. It seems you have a good knowledge of the API plus being a live user is extremely valuable.
6 Views
Open in Slack
PreviousNext
Powered by

Meltano

Meltano community

Powered by