could someone help me with setting up a docker compose for meltano? or tell my why the included files dont work. i’m running macOS big sur and have dificulties installing it natively so i just wanted to use docker like i normally do for all my project but i cant get the included files to work. it already says

Error: No such command "bash".

and the file i’m using is https://gitlab.com/meltano/meltano/blob/master/docker-compose.yml. doesn someone have any ideas?

Have you been able to do similar things from the command line in Big Sur? There was an HN thread about a number of changes, including xcode updates that need to be manually triggered

i have never gotten it to work because i always have been on big sur since i found out about meltano. but i used standard docker but i’m way more comfortable in docker-compose and find it easier to setup database that way, have it all in 1 command. also when i tried to use meltano commands in docker to create docker compose file it didnt create it but that is prob because it created in the docker container not the directory i was in

@joy_timmermans Instead of https://gitlab.com/meltano/meltano/blob/master/docker-compose.yml, I recommend you use the

docker-compose.yml

from the

docker-compose

file bundle: http://meltano.com/docs/ui.html#docker-compose https://gitlab.com/meltano/files-docker-compose/-/blob/master/bundle/docker-compose.yml

also when i tried to use meltano commands in docker to create docker compose file it didnt create it but that is prob because it created in the docker container not the directory i was in

Did you have your project directory mounted when you added the file bundle? It would look something like this:

docker run -v $(pwd):/project -w /project meltano/meltano add files docker-compose

That would add

docker-compose.yml

to your current working directory (

pwd

), which is mounted into the container at

/project

.

thanks for giving that other link. but that docker compose doesn’t allow you to run

docker-compose run --rm meltano --version

for example? adding a database container i can do. but why not give one environment that does it all out of the box?

That

docker-compose.yml

gives you an environment appropriate for development, where any changes you make in the UI end up in

meltano.yml

in your project dir, and where a system database is not necessary. For a more production-like setup, there's https://meltano.com/docs/containerization.html#docker-compose and https://gitlab.com/meltano/files-docker-compose/-/blob/master/bundle/docker-compose.prod.yml, which uses the project-specific Docker image that doesn't allow any changes to

meltano.yml

anymore (since they wouldn't be persisted outside the container), and uses a separate system database. In both setups, the service is called

meltano-ui

rather than

meltano

(so you'd need

docker-compose run --rm meltano-ui --version

) to make it clear that the UI does not represent all of Meltano, and that that service is not responsible for actually running Meltano ELT pipelines, which is where the

airflow-scheduler

service comes in. I think in your case, it would be easier to have a single

meltano

service that runs both the UI and the scheduler, that could then easily be used through

docker-compose run --rm meltano <args>

. This would require a way to tell

meltano ui

to also run the orchestrator/scheduler, using an env var or CLI argument. This was recently discussed in https://meltano.slack.com/archives/CFG3C3C66/p1603820355263900?thread_ts=1603627483.239000&cid=CFG3C3C66, if you'd like to have some more context. If you think this would be useful in Meltano and simplify things for you and new users, do you think you could create an issue? 🙂

aahh i see. i did know the ui was just a webserver. but tought there was just a cli program for the tasks (and some extra/more robust stuff) that are handeld in the ui. what does the scheduler do then? is there some documentation on how the program works? if it still unclear i will gladly make a issue Also that you say there is no system database necesary how do you create a dashboard then? you need a extractor and a leader? but the loader is a database? correct me if where I am wrong? I really like what the project could bring but it seems a little hard to grasp at first for a new user. i think what would really help is making a small youtube vid where you setup Melton from a clean install to a dashboard and explain some of the things why it is that why along the way. just a short 5-10 min to get new people going.

what does the scheduler do then?

The scheduler is the part of the orchestrator (Airflow in our case) that's responsible for automatically running the scheduled pipelines at the defined intervals: https://meltano.com/docs/orchestration.html If you're not using an orchestrator and its scheduler, you have to manually set up some system (like cron) to automatically run

meltano elt <args>

whenever you want it.