Hey guys new to singer tap development here I just configure Meltano #singer-tap-development

Hey guys, new to singer tap development here. I ju...

zlatan

05/09/2023, 2:46 PM

Hey guys, new to singer tap development here. I just configured custom rest tap and postgress target, and everything is working correctly. I’m just having difficulties with state. In my REST api, there is a field called fromDate, and I can query records from the given date. I defined that date in my streams.py file:

replication_key = "purchaseDate"

That state is saved correctly, and I have the correct value there: state.json file:

"bookmarks": {

"purchases": {

"replication_key": "purchaseDate",

"replication_key_value": "2023-05-09T13:34:20.000Z"

The question is now, how can I use

replication_key_value

and send it as query param to my rest in here:

params: dict = {}

if next_page_token:

params["page"] = next_page_token

if self.replication_key:

params["fromDate"] = # get replication_key_value

return params

josue_sehnem

05/09/2023, 2:52 PM

you can use the

get_starting_timestamp

method something like

self.get_starting_timestamp(context)

should return you the state for the stream and you can use it as the parameter. I usually also add a

start_data

parameter to the tap and use a function like:

Copy code

@cached
    def get_starting_time(self, context):
        start_date = self.config.get("start_date")
        if start_date:
            start_date = parse(start_date)
        rep_key = self.get_starting_timestamp(context)
        return rep_key or start_date

josue_sehnem

05/09/2023, 2:53 PM

I think that there is a build_in method that do the same thing too, I am just not remembering it.

Andy Carter

05/09/2023, 2:57 PM

I think get_starting_timestamp
handles the

self.config.get("start_date")

bit too, at least according to the docs: https://sdk.meltano.com/en/latest/classes/singer_sdk.Stream.html#singer_sdk.Stream.get_starting_timestamp

zlatan

05/10/2023, 11:39 AM

Thanks guys, this resolved my issue! Appreciate it

Open in Slack

Previous Next