Hey Meltano Team, Do we have any way to ask Meltano continue with data downloads even if there is failure downloading one of the objects? For example if we are downloading 100 objects using tap-salesforce and one of them failed due to any reason the entire process is stopped and a critical error is raised. I am looking for some solution to continue downloading and handle the error later, if it is non critical object.

Hi @rahul_anand! If the tap itself is raising the error and failing, there's nothing Meltano can do to prevent that. If some errors should be ignored, that would need to be implemented in the tap itself.

Hi @douwe_maan, As an orchestration platform, I guess, it would be good for Meltano, to provide some mechanism of retrying failed data downloads and continue processing after one object failure. In my view a

tap

is created for generic use case so it may be ok for tap to flag as error and stop, but the orchestrator has a bigger responsibility to implement different workflow requirements of various use cases. Can Meltano have some intelligence with its state data to understand re-try configuration and try few times before giving up? And also continue downloading other objects excluding the failed object.

I agree, let’s see if Meltano can be smarter here! The simplest thing we could do is add a

retry: 3

setting to have

meltano elt

retry itself (or just the tap) just in case a failure was recoverable, but that wouldn’t help if one object was fundamentally broken and in downloadable. I’m not sure how Meltano could detect a broken object and then specifically decide to skip it (or its entire stream), though, since it wouldn’t know what caused the tap to fail, not even the specific kind of object (stream) it was trying to download... How about we explore either or both some more in an issue? :) Would you mind creating those, with a description of your use case and specific problem, and any thoughts you have on how we might implement these or any other potential retry strategies?

@douwe_maan I am still struggling with this issue so created a Jira issue: https://gitlab.com/meltano/meltano/-/issues/2436. For now we have to write quite complex Airflow logic to detect failed objects and then use

--select

to pass failed object and remove

--full-refresh

flag, on retry of the

elt

pipeline. A retry feature will be really helpful.

Thanks @rahul_anand, I'll respond on the issue!