Hi everyone, I recently started working with Meltano and Docker, but I’m running into some issues. I have an urgent project where Airflow needs to act as the orchestrator, and Meltano will handle the extract and load (EL) processes. The data will be extracted from a PostgreSQL database, and the entire setup must run using Docker. I’m using the docker-compose file provided by Airflow as a template. However, when I add a Meltano container to the docker-compose configuration, the Meltano container keeps restarting (not sure if that’s intended). Additionally, I’m unable to execute Meltano commands in an Airflow DAG using the BashOperator (gives an error "meltano command not found"), which is a key requirement for my project. I’ve spent over 20 hours searching for a solution, but as a beginner, I might be missing something fundamental. I’m feeling very lost and could really use some help! Where do I start to effectively use Docker, Airflow, and Meltano at once, while building this connection that enables me to run Meltano commands with BashOperators? I'm sorry if this is an unrelated question, I'm very lost and starting to lose hope.

You should use this doc it's quite straight forward. I'd advise you build it bit by bit before jumping into orchestration.

Hello! I’ve made progress and was able to run Meltano without any problems—thanks for the help. I haven’t started orchestrating yet, but I have a question. Is it possible to remove the schema name from the table when extracting data using

tap-postgres

and

target-csv

? My outputs are currently in the format:

schema-name-table-name.csv

. I’d like to have the output in the format

table-name.csv

instead. Additionally, is there a way to dynamically include the extraction date, such as

2024-01-01/table-name.csv

?

You will have to use mapper for it and you find some examples in this SDK doc and this as well

Thanks a lot! I researched it and was able to implement it. Is there a way to dynamically configure the output path in

target-csv

? For example, to include the table name, date, and other variables in the path. Or is this too complex to implement?