Hi everyone First of all thank you all for your help Current Meltano #singer-tap-development

Hi everyone, First of all, thank you all for your...

alexandre_brehelin

09/10/2022, 11:47 AM

Hi everyone, First of all, thank you all for your help. Currently, I'm trying to develop a custom singer tap with aircall api. I'm using first endpoint

/v1/teams

to fetch all teams then I call

/v1/teams/:id

endpoint to retrieve details for each Team. To do that I'm using parent-child streams like below where aircallStream is RESTStream object.

Copy code

class TeamsStream(aircallStream):
    name = "teams"
    path = "v1/teams"
    primary_keys = ["id"]
    replication_key = "created_at"
    schema = teams_properties.to_dict()
    records_jsonpath = "$.teams[*]"

    def get_child_context(self, record: dict, context: Optional[dict]) -> dict:
        return {"team_id": record["id"]}

class TeamStream(aircallStream):
    name = "team"
    parent_stream_type = TeamsStream
    path = "v1/teams/{team_id}"
    primary_keys = ["id"]
    schema = teams_properties.to_dict()
    records_jsonpath = "$.team[*]"
    state_partitioning_keys = []

At first run, the tap retrieve all data for each team and create state json file. Then, when I used the state file the tap seems not to use the bookmark. I retrieve all data like first run. The

/v1/teams

endpoint not take date params, so we can't use

self.get_starting_timestamp(context)

get_url_params

. So, I guess at each time
TeamsStream
send all teams to
TeamStream
but in RESTStream are there a method which filter child according to bookmark replication_key ? More clearly, to use replication_key and replication method we must have a stream where the endpoint take replication_key as params ? Last question about parent-child streams, with class above, Can you confirm me that the parent stream calls its endpoint once and not as many times as there are child streams? Thanks.

edgar_ramirez_mondragon

09/12/2022, 8:55 PM

Hi @alexandre_brehelin!

in RESTStream are there a method which filter child according to bookmark replication_key ?

Not a sanctioned or explicit method to filter out older records, and it’s not documented but can still be done using post_process. See this note: https://gitlab.com/meltano/sdk/-/issues/227#note_926360644 (github issue: #226).

Can you confirm me that the parent stream calls its endpoint once and not as many times as there are child streams?

Yes, the former. Parent stream is called once and child stream is called once per parent record.

alexandre_brehelin

09/13/2022, 7:20 AM

Thank you Edgar, that is very clear. I will use the post_process method 🙂

Open in Slack

Previous Next