Hi, can someone please help how should I configure tap-freshdesk so that it would extract all comments, not just the initial description of the issue? Thanks.

@Andy Carter I think Eli is using your variant if you have any insights here!

@eli_c. can you confirm which variant you're using? Which api path would you like to hit for comments? Do you mean the

conversations

endpoint? The tap should already be configured to give you that. If there's another endpoint in the docs (https://developers.freshdesk.com/api/) you would like then let me know. Can you share your meltano.yml and I'll take a look.

Hi @andy_carter, I am using

acarter24

variant. Yes the conversations, for now with the default configuration I only get the initial message, no responses or replies.

version: 1
default_environment: dev
project_id: 88ce441c-e6a8-4af8-9bd8-20d220e34a4d
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-freshdesk
    variant: acarter24
    pip_url: git+<https://github.com/acarter24/tap-freshdesk.git>
    config:
      domain: fd2j
      start_date: '2023-05-20'
  loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl

Are you seeing the

conversations

endpoint being synced when you run?

yes

And any jsonl output for that stream? Does that not give you the data you want? You should see each conversation has a linked parent ticket id.

correct, the jsonl file itself looks fine, just doesn’t have the entire conversation. also attachment and embedded url missing (could be a different issue?)

Can you give some examples as to what is missing? Can you make a postman call and show the missing data that the tap isn't getting?

the jsonl file of one issue:

{
  "cc_emails": [],
  "fwd_emails": [],
  "reply_cc_emails": [],
  "ticket_cc_emails": [],
  "fr_escalated": false,
  "spam": false,
  "email_config_id": 150000090571,
  "group_id": 150000274008,
  "priority": 1,
  "requester_id": 150023570532,
  "responder_id": 150023158907,
  "source": 1,
  "company_id": 150000698050,
  "status": 6,
  "subject": "How to download the abc",
  "association_type": null,
  "support_email": "<mailto:support@fd2j.freshdesk.com|support@fd2j.freshdesk.com>",
  "to_emails": [
    "<mailto:support@fd2j.freshdesk.com|support@fd2j.freshdesk.com>"
  ],
  "product_id": 150000047073,
  "id": 4,
  "type": null,
  "due_by": "2023-05-30T21:00:00Z",
  "fr_due_by": "2023-05-26T21:00:00Z",
  "is_escalated": false,
  "description": "<div dir=\"ltr\">\n<div style='box-sizing:border-box;word-break:break-word;color:rgb(24,50,71);font-family:-apple-system,\"system-ui\",\"Segoe UI\",Roboto,\"Helvetica Neue\",Arial,sans-serif;font-size:14px'>Hi!</div>\n<div style='box-sizing:border-box;word-break:break-word;color:rgb(24,50,71);font-family:-apple-system,\"system-ui\",\"Segoe UI\",Roboto,\"Helvetica Neue\",Arial,sans-serif;font-size:14px'><br style=\"box-sizing:border-box\"></div>\n<div style='box-sizing:border-box;word-break:break-word;color:rgb(24,50,71);font-family:-apple-system,\"system-ui\",\"Segoe UI\",Roboto,\"Helvetica Neue\",Arial,sans-serif;font-size:14px'>We have a control center license and want to download the latest version of control center for the license, how do I do it?</div>\n<div style='box-sizing:border-box;word-break:break-word;color:rgb(24,50,71);font-family:-apple-system,\"system-ui\",\"Segoe UI\",Roboto,\"Helvetica Neue\",Arial,sans-serif;font-size:14px'><br style=\"box-sizing:border-box\"></div>\n<div style='box-sizing:border-box;word-break:break-word;color:rgb(24,50,71);font-family:-apple-system,\"system-ui\",\"Segoe UI\",Roboto,\"Helvetica Neue\",Arial,sans-serif;font-size:14px'>Regards</div>\n<div style='box-sizing:border-box;word-break:break-word;color:rgb(24,50,71);font-family:-apple-system,\"system-ui\",\"Segoe UI\",Roboto,\"Helvetica Neue\",Arial,sans-serif;font-size:14px'>Summer Time</div>\n</div>\n\n",
  "description_text": "Hi Summer,\r\n\r\nMy appologies, that was a bad link.\r\n\r\nHere you go: download here.\r\n\r\nThanks,\r\nEli\r\n\r\n",
  "custom_fields": {
    "cf_reference_number": null
  },
  "created_at": "2023-05-26T04:04:02Z",
  "updated_at": "2023-05-26T04:09:37Z",
  "tags": []
}

Can you definitely see the conversations via the freshdesk api? Try this

curl -v -u yourapikey:X -H "Content-Type: application/json" -X GET '<https://fd2j.freshdesk.com/api/v2/tickets/4/conversations>'

then which part of output i should look at? what am i looking for here? sorry, i see the whole conversation is there.

Are you looking at the abridged or detail ticket stream for the attachments? Attachments will only show in 'tickets_detail' stream. I don't really understand what your images are showing me here, sorry.

tickets_detail

Ah ok I was trying to keep the payload light so I may have excluded attachments

Enjoy!

Is the only thing you're missing the attachments from each conversation item then? You mentioned some hyperlinks but I think it might the freshdesk API sanitising those, there certainly isn't any cleansing in my tap code that would do that.

three things: 1. conversation kept only the initial one in jsonl (even though i can see from the call) ; 2. attachments from each conversation; 3. yes the hyperlinks. if it’s not cleansed by your tap, any suggestions on how to go about restore the links? in addition, when using target-csv the format will be messed up. it’s perfect with target-jsonl.

Ok maybe we can set up a call next week to resolve. In the meantime could you try the stitch variant and see if you get the same behaviour?

Sounds great. I will try that and let you know the result. Happy long weekend.

in addition, when using target-csv the format will be messed up. it’s perfect with target-jsonl.

@eli_c. this seems like it could be a different issue, potentially related to target-csv or the output format of the tap but it might be worth using a different thread for that since theres already a lot going on in this one 😄

Another suggestion, if you run 'meltano invoke tap-freshdesk' do you see any more of the missing conversations? Just wondering if jsonl is the culprit.

could be. the conversations are there.

Maybe try 'meltano run tap-freshdesk target-jsonl --full-refresh' and see if the state is the issue?

Maybe try ‘meltano run tap-freshdesk target-jsonl --full-refresh’ and see if the state is the issue?

This didn’t make any difference. Trying sqlite now.

I'll check with the attachments, sure. On the hyperlink issue, can you make a curl/postman request and see if the hyperlink url is visible there and share the JSON here or via DM? If the link isn't in the raw JSON response there's nothing I can do. Edit: Ok I can see the conversations endpoint returns both body_text which is plain text, and 'body' which is html and would contain links. The second key is missing from the schema json. I'll try for a fix next week

@eli_c.`variant` could you test this branch? I think you can do it by adding the branch name to your

pip_url

and then doing

meltano install --clean

.

sorry was caught up with other work.

try this!

git+<https://github.com/acarter24/tap-freshdesk.git@1-add-new-fields-to-conversations-stream>

Thanks @pat_nadolny install worked, but then tried all three loaders I have installed, all failed with the same msg :

Loader failed

I've lost the context in this thread. Can you reshare the error traceback? Heres some suggested troubleshooting steps too https://docs.meltano.com/guide/troubleshooting#log-level-debug

objective: extract issues from freshdesk, load into csv or whatever that can be imported to Jira. issues came up: following information missing - 1. conversation stream, i.e. only the initial message was loaded into the target output; 2. embedded hyperlink; 3. attachments. diagnose: by making curl/postman requests, we can see some of the missing information were tapped. so I assume the issues were not with the tap.

ok that helps. I think we should attack one problem at a time. Where are you seeing the

Loader failed

message? I would suggest not using a loader yet to isolate the tap, run

meltano invoke tap-freshdesk > output.json

to confirm that the tap runs successfully and your data is there. Then we can move on to isolating the target. We dont want a target issue to cause confusion with Andy's PR to get your new attributes added

Where are you seeing the

Loader failed

message?

After changed

pip_url

in yml, i ran

meltano run tap-freshdesk target-jsonl

(and

target-csv/target-sqlite

), all three loaders got the same error in the end:

2023-05-30T19:30:20.306661Z [error    ] Loader failed
2023-05-30T19:30:20.306850Z [error    ] Block run completed.           block_type=ExtractLoadBlocks err=RunnerError('Loader failed') exit_codes={<PluginType.LOADERS: 'loaders'>: 1} set_number=0 success=False
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Run invocation could not be completed as block failed: Loader failed

And if you run

meltano invoke tap-freshdesk > output.json

what do you see?

Also usually if you scroll up in the traceback that you shared it will eventually tell you the actual reason why the loader failed

i see no error, but still trying to make sure that no data is missing.

thats a good sign!

the output does not give enough details that you can be sure all the data is there, right?

the output does not give enough details that you can be sure all the data is there, right?

All the tap data should be in that output.json

a few lines back:

thats getting closer! It looks like maybe an index error where some data is missing but if you scroll even higher it should give you more detail.

What you can do when you're happy with the data in output.json (i.e. meaning your done reviewing Andys recent changes) you can run

cat output.json | meltano invoke target-sqlite

to load that file of data into sqlite. This is identical to the process of running

meltano run tap-freshdesk target-sqlite

so it will isolate the problems

trying these suggestions now. Thank you very much @pat_nadolny

@eli_c. ahh that might be it! Try manually deleting the

.meltano/run/tap-freshdesk/

folder in your project. I think you might have a cached catalog file from the older version of the tap before you reinstalled

you are right. mind me asking how did that error tell you that?

that warning message is saying that its seeing the attachments property in the actual data being synced but its not in the catalog so the properties are getting filtered out. Basically the workflow that meltano does with singer taps is 1.) asks for a catalog of all available streams and properties 2.) alters that catalog based on your meltano.yml configurations 3.) passes that altered catalog to the tap when its time to sync the data. In your case meltano was caching the old catalog so even though the tap had new properties meltano never did step 1 again to get a refreshed catalog

check out https://hub.meltano.com/singer/spec if you want to learn the internals of how singer works (meltano uses singer under the hood)

sure want to learn that.