Hi! 1. Is it possible to run an airflow schedule, but to start not from the last completed day, but from the start_date?

@zlatan Do you mean that you'd like to always extract all records after start_date again and again each time the pipeline runs, without trying to pick up where the last run left off?

Not always, but to schedule a data load refresh from the initial start date for example every 6 months.

@zlatan On an individual

meltano elt

run, you can use

--full-refresh

to ignore the state and do a full refresh, so what you're looking for is a way to run that command on a 6-monthly schedule, but we currently don't support

--full-refresh

on

meltano schedule

or

full_refresh: true

in

schedules

in

meltano.yml

. Would you like to create an issue for that? 😄

Yes, sure 🙂

@zlatan This would actually be a nice one for you to contribute if you're up for it 🙂

Great, I'll look into it 🙂

Cool, I'll leave some suggestions on the issue once it's created!

--full-refresh

replaces the data in the target, correct? it wouldn't create duplicates?

@zlatan That's up to how the target is implemented. It likely uses the tables' primary keys to update the records that already exist instead of inserting them anew.

Great