hello 👋 I have a composable pipeline that looks more or less like this:

meltano run tap-postgres-sync-base target-s3-jsonl tap-postgres-sync target-redshift --full-refresh

. How would I set this up to have the full-refresh applied to the second part only? Ie, id like to have

tap-postgres-sync-base target-s3-jsonl

running incrementally and

tap-postgres-sync target-redshift

ignoring the existing state… right now full-refresh is applying to both components 🤔

today meltano run args like full-refresh only apply to the entire meltano run invocation. So the only way to do it would be to break it up into two distinct invocations.

If there is dependency between them:

meltano run tap-postgres-sync-base target-s3-jsonl && meltano run tap-postgres-sync target-redshift --full-refresh

@elena_velte - As noted above, this isn't supported as of now, apart from the workaround from @alexander_butler. Can you say a bit more about your use case?

thanks for helping out 🤗 I’m using this for a weekly hard reload; so yes, I usually do want to track state, and only sometimes do a full refresh. what I came up with is something similar as suggested above, where I simply drop the state for the second part before running the “full-refresh” run. that would look like

meltano --environment meltano-run state clear --force meltano-run:tap-postgres-sync-to-target-redshift && meltano --environment meltano-run run tap-postgres-sync-base target-s3-jsonl dbt:tiger-sync-cleanup tap-postgres-sync target-redshift dbt:test

what would you say would the best practice be for this use case?

@elena_velte - Yes, I do think clearing the state ahead of time is a really good approach and should produce the affect you are looking for with a hard reset. Optionally, instead of fully clearing state, you could also rename or move it, or print it to a log with the

get

command, just in the case of a failed load or some other issue. Either way though, I think the above makes sense. 👍

nice, really appreciate your input! thank you ☺️ 🙏