Hi everyone, just picked up meltano and I'm trying to figure out the best way to do this. I have successfully ran a tap-postgres target-csv which creates one file for each table like in the image. Now I have to run a tap-csv target-postgres, and everything must be orchestrated. I'm trying to understand if it's possible to do it with a single meltano el tap-csv target-postgres. The issue I'm facing is that I don't know the names of the files preemptively because of the date that gets appended . I read the docs about orchestrating with airflow and it seems I have to define jobs, and I don't know if it's possible to make one job per file as it would require me to get the filename and change the config of the tap-csv on the fly. So to coalesce into a single job I tried to use wildcards but doesn't seem to work

path: loaded_data/order_details-*.csv

. I'm just trying to understand what command(s) the job should run in this case before I even install airflow

Could you just run

meltano run tap-postgres target-postgres

in terms of practicality yes it makes more sense but it's just a requirement from someone

Have you taken a peek at https://hub.meltano.com/loaders/target-csv--meltanolabs/#file_naming_scheme-setting , I like to read them from the repo's README here https://github.com/MeltanoLabs/target-csv#:~:text=be%20created%20automatically.-,file_naming_scheme,-False https://github.com/MeltanoLabs/tap-csv , then point to each of the stream names individually, either by updating that file dynamically or the config dynamically before the job. Defineitly possible to do before the job runs, you can use plugin inheritance or if you want to be dynamic and you could set an environment variable before the run, or run

meltano config

before the run to update the location you want to pull the file from based on the names.

it seems i forgot to take a look at other variants, thanks i'll check it out