Is there any way to disregard the

start_date

of a stream when there's no state yet (using the SDK)? I have a project where I don't know when the first record was created and need to fetch all data for the first run (and then incrementally fetch subsequent runs). Obviously I could use a date way back in history and hope it includes all data, but that doesn't feel right. Any tips?

There’d be a couple of ways to solve this. You can use an

environment

to override the start date for a single run - downside of that is that you’d have to copy over the state to run in a new environment. I belive you can also use an environment variable to override the config - should be something like

<plugin_name>_<setting_name>

Thanks for the tips. I've actually just included an extra if-statement in the stream, because the timestamp had be excluded from the

url

as well. It's not the best solution, but it does the trick for now

You could just make the

start_date

nullable. The methods used to retrieve the stream bookmark will 1. try to use the state 2. Fall back to

start_date

if present 3. Return

None