The restrictions I have: 1. A

tickets

endpoint, which returns a summary of ticket info, 100 tickets at a time. I need to paginate through these quickly and get a list of ticket `id`s, otherwise tickets get missed in the pagination. 2. A ticket detail endpoint

ticket/{iid}

that gives the full detail, plus access to child conversations endpoint

ticket/{iid}/conversations

3. A very restrictive 429 error, which means I am allotted about '100 call credits' a minute, each ticket and related conversation eats up around '5 calls' I would like to 1. Run

tickets

as a normal stream, with state, to get me all the

id

s that have changes. Lets call this

TicketSummary

stream. 2. Access the IDs generated in a run of

TicketSummary

stream from an instance of

TicketDetail

stream, which accesses

ticket/{iid}

and

conversations

in turn at a much slower rate. Why can't I do it in the normal way? • The low head room on calls per minute means I can only get a few tickets at a time before being forced to wait 50s ish. • This gives lots of time for tickets to be updated by agents 'during' a run. • Which causes a messup in pagination, and some tickets which should have been included to get missed in the sync.

I have tried to use a

set

in the parent Tap to collect the relevant `id`s, but it looks like

get_records

is being called / the generator set up for each stream set up first, and then being iterated later.

OK made some progress here, needed to realise that the order that streams are defined on the tap don't dictate the execution order. I can force one stream to run before another, by making one a child of the other. Then sharing a set works, the parent stream runs and fills the set with Ids, and the child stream comes in and reads them running queries.

@Denis I. any idea why I'm getting this error

meltano select --list --all tap-freshdesk

Cannot list the selected attributes: Could not find catalog. Verify that the tap supports discovery mode and advertises the `discover` capability as well as either `catalog` or `properties`

I can't get any

select:

filters to work either. What am I missing? https://github.com/acarter24/tap-freshdesk

I will investigate some more, thanks for the pointer

Looks like if I add these capabilities to

meltano.yml

that command gives a proper response.

- name: tap-freshdesk
    pip_url: git+<https://github.com/acarter24/tap-freshdesk.git>
    capabilities:
      - discover
      - properties
      - state

Why would that help though? Is it something to do with a version of

tap-freshdesk

already existing on the hub?

EDIT: Adding this block:

capabilities:
      - discover
      - state
      - catalog
      - about
      - stream-maps

Made everything work much better

Actually my use case is a bit more complicated perhaps: need to access a few streams that will be used to paginate other streams with: countries, cities, categories,… and I don’t think streams can have multiple parents. But this was a great start