Is there a way to configure an end_date on a run for a tap that doesn't directly support it in its documented settings? We're attempting to replicate 8 years of historical data from Square, but without an end date, we're having difficulty finding a period of time where we can let this process run long enough to complete.

You would likely need to add that capability to the tap itself. There’s no native way within Meltano other than do filter the data out after it’s been extracted - but that doesn’t sound like it would solve your problem.

That's what I figured. Thanks for confirming @taylor!

but if the API supports an

end_date

parameter, then the tap should definitely be updated to enable that for everyone. so if you do fork it and add that capability please consider making a PR upstream 😄

@chris_schmid one of the beautiful things about state is it should get saved along the way allowing you to stop processing and start where you left off without having to use an end date! That's where I'd start atleast 🤷

@visch That's where we want to be, but right now our state is transient with each run because we're running it through Docker and don't have it hooked up to an external system db yet. We use a bash script to tweak the start date to yesterday during each run to pull in current data (I know, not the best idea). Meltano/Docker are both very new to our organization and we've been trying to onboard under a time crunch. But you raise a good point, and convinced me that I need to spend the time setting this up properly with a backend db to manage metadata.

😄 I understand! https://docs.meltano.com/concepts/state_backends#undefined Might be even easier if you have s3 available! If you're running on a docker container that's local and can mount a directory that could work too (.meltano dir or https://docs.meltano.com/reference/settings#database_uri and point to a mounted sql lite DB file )