https://linen.dev logo
#singer-tap-development
Title
# singer-tap-development
h

hawkar_mahmod

03/22/2023, 12:54 PM
Hey folks, Would appreciate some guidance on how to progress with my Singer tap implementation. If you look at my previous message above this one you'll see I'm building a REST based tap. I'm not sure how to deal with state in my situation. The data I am pulling is aggregated report data which is keyed on a Date and some other columns. I'd like to run my pipeline once per day and retrieve data for the past 7 days so I can backfill values and account for late arriving data. As I understand, the SDK doesn't offer a way of dealing with this use case directly. Since there is no single replication key using the default SDK interface doesn't work for me. Instead I need to manually work with the state object to store the last run date and derive my start/end dates based off of this. IMO this is the most robust approach and ensures that at the target I overwrite records based on a combination columns I designate as a my primary key. Alternatively I can avoid state altogether and just set the start and end dates for the API based on config values which I pass via env vars. If the first approach, using State, is best than can you advise on where to start with this in terms of implementation. Just a pointer in the right direction would be very helpful.