Can someone please help me understand the best way to configure Meltano to automatically harvest data from a tap historically (i.e. today - 7 days) on an daily basis? Basically we need to allow for up to 7 days for advertising data sets to reach 'eventual consistency' and collect a single day's worth of data.
I was figuring the start_date environment variable needs to be programatically controlled for a tap/schedule - just I'm not sure the best way to do that to ensure it's not baked into the docker container
04/14/2021, 2:43 PM
Hi, @creamy-lock-2197! If the tap you are using supports incremental updates, then each record will have an internal update indicator (generally an updated-at time) and that will automatically capture those new updates to reach their fully consistent and populated state.
A long way to say it, but if I properly understand your question, I think the behavior you want should be automatic without having to modify state or the start_date.
If the tap does not support incremental, then a workaround would be to update the start date input as you described. Writing the dynamic start date calculation is possible but varies depending on your deployment mechanism.