Good morning everyone! I'm just a newcomer to the Meltano ecosystem, looking for some practical hints. As far as I know, there are several extractors to integrate files like the SFTP taps, to extract files into JSON. My question is about the stages before loading the data itself. Is there any best practice regarding file consolidation (copying/moving files)? Is that a task directly delegated to Airflow or any other orchestation software?

Yes. The meltano

elt

command is a wrapper around taps and targets. So the taps theoretically could do anything, including moving files or other prep work. But typically they are much more focused, only extracting data from the source system like SFTP.

Thanks Adam for your response! I will check Prefect too, as it sounds quite interesting. Regarding moving files, I've seen most operators just copy, without ever deleting. Do people resort to rsync/mv or custom python code to check for successful data transfer before deleting on source?

Hmm I think deleting from the original source is not a common practice. More common is to track some sort of key like a timestamp, and filter each extraction to only get the latest/changed records.