How do you reset state in meltano (tap-salesforce) to get it to do a full load again?

I do this by deleting the records from jobs table in meltano systems db.

meltano elt

actually accepts a

--full-refresh

flag to do just that!

This is great @prratek_ramchandani! I think full-refresh option should also be added to meltano ui next to the

Run Now

button. Also files-airflow bundle should accept a full-refresh parameter via configuration json from airflow ui which can be used by the DAG run using {{ dag_run.conf }} to do a manual full refresh. What do you think @douwe_maan, @taylor? The files-airflow bundle change looks quite simple to me and I can add an MR. Having an option to run full-refresh via a UI (meltano or airflow) is important to me because I am running meltano on ECS fargate and don’t want to allow ssh access. And I am currently doing a full refresh by deleting jobs record from meltano system db,

@pankaj_saini go for it! If you don’t mind opening an issue first to coordinate any discussion that’d be great. But we love MRs 🙂

Amazing! I will do that. I will also raise an issue for a full refresh button on meltano UI if there is not an existing issue.

I’m not aware of any but to be truthful I haven’t read through every issue yet 😅 still splitting my time between Meltano and the data team (though I’m focused on Meltano for the next 3 business days 🎉 )

There is an existing issue for the meltano UI. I will add one for files-airflow bundle.

should be a meltano cli command that wipes the state without having to do a run (if you want to be orthogonal about it)

That's a really good point @chris_kings-lynne. But for UI probably one of the button should be enough, i.e. either Clear State, or Full Refresh. But for CLI it would be nice to have an option to clear state without meddling with system db externally.

@chris_kings-lynne https://gitlab.com/meltano/meltano/-/issues/2568