Hi Can anyone help me to implement incremental data pull for Meltano #singer-taps

Hi Can anyone help me to implement incremental d...

Sekhar Baggaraju

04/10/2024, 6:46 PM

Hi Can anyone help me to implement incremental data pull for tap-freshdesk tap . I added in yml file as below but didn't work.

Copy code

settings:
- name: username
- name: password
  kind: password
  sensitive: true
- name: start_date
  value: '2024-04-15T00:00:00Z'
config:
  start_date: '2024-04-15T00:00:00Z'

Edgar Ramírez (Arch.dev)

04/10/2024, 8:26 PM

What does the rest of the plugin definition look like?

Sekhar Baggaraju

04/11/2024, 7:56 AM

it has creds and target . I am trying to find a place to provide datetime so that tap-freshdesk can pull data from that start date onwards

Andy Carter

04/11/2024, 8:05 AM

Can you provide more information when you say 'doesn't work'? Please share the whole

meltano.yml

file not just a section.

👍 1

Andy Carter

04/11/2024, 8:08 AM

The incremental piece should work using the saved state once you have completed a full run, I use it in production.

Andy Carter

04/11/2024, 8:09 AM

Here is my meltano.yml

Copy code

- name: tap-freshdesk
    capabilities:
    - discover
    - state
    - catalog
    - about
    - stream-maps
    config:
      domain: ${TAP_FRESHDESK_DOMAIN}
      start_date: 2024-04-01
      api_key: ${TAP_FRESHDESK_API_KEY}
      embeds:
        tickets_detail:
        - stats
        - sla_policy
      stream_maps:
        contacts:
          email: md5(email)
          name: md5(name)
        conversations:
          body_text: body_text[:100]
          to_emails: __NULL__
          cc_emails: __NULL__
          bcc_emails: __NULL__
        tickets_abridged:
          to_emails: __NULL__
          cc_emails: __NULL__
          ticket_cc_emails: __NULL__
        tickets_detail:
          to_emails: __NULL__
          cc_emails: __NULL__
          ticket_cc_emails: __NULL__
          description: __NULL__
          description_text: description_text[:100]

Sekhar Baggaraju

04/11/2024, 8:38 AM

Thanks for sharing. can you share example for state save approach ,it will be usefull

Andy Carter

04/11/2024, 8:44 AM

You don't need to do anything else to achieve state save, it will happen automatically with meltano taps that support it. https://github.com/singer-io/getting-started/blob/master/docs/CONFIG_AND_STATE.md#state-file

Sekhar Baggaraju

04/11/2024, 9:09 AM

Thanks Andy. It seems like tap-freshdesk doesn't support state save.

13 Views

Open in Slack

Previous Next