Hey team, Revisiting the project after a few months on the sidelines Question: The docker instances systems, is there a fully turnkey version of docker? Over the weekend I was trying to sort a few things from our Cyberpanel DO droplets which the team is trained on, only the docker required further cli config after launching. Looking for a means where one of the team members can simply create a new environment or project per client and then start configuring without touching the backend systems. Looking at 2 options 1. Figure out how I can easily manage multiple 'projects' / client on the 1 installation, ie domain.com/project1 domain.com/project2 etc 2. Setup a docker instance per client, where the client would be the name of the docker container and everything within the image would be preconfigured. In the documentation, I can see the reference to 'Project', how I haven't seen any documentation on how you can access each project where you have multiples. I can see that the loaders have had some development since I last checked, but I am a little lost on the configurations and I am unsure if it still requires a localised DB as localhost was not editable, documentation was a bit thin on this https://meltano.com/plugins/loaders/postgres.html Objective - Wanting to centralise all the connections into a central DB which would become problematic if we are running multiple docker instances per client. With the docker, is there a variation that would auto-configure a turnkey default project? and if it has to run a localised database, then preconfigure all the required dependencies. Thus allowing a click n play docker setup

Hi @dwayne! Are your team members comfortable with Git and using CLIs? Meltano is currently optimized for users who are, per https://meltano.com/docs/#focus:

They are comfortable using a CLI and want to manage their Meltano project like a software engineering project, with all of the benefits of version control and CI/CD.

It sounds like you've been evaluating the Meltano UI, which is not currently our priority. It has a number of limitations and missing features that are tracked in https://gitlab.com/groups/meltano/-/epics/78, but we're not currently allocating time to these. The UI is still receiving regular contributions and improvements from the community, and we'll gladly welcome more, but I expect the UI experience to be subpar to the CLI experience for the foreseeable future, so if it at all possible, I'd suggest looking into using the CLI.

1. Figure out how I can easily manage multiple 'projects' / client on the 1 installation, ie domain.com/project1 domain.com/project2 etc

This is not currently possible: each Meltano project currently needs a dedicated

meltano ui

webserver, and there is no UI for administering (creating, deleting) projects. If you'd like to see this change, I invite you to create issues on https://gitlab.com/meltano/meltano/-/issues 🙂

I am unsure if it still requires a localised DB as localhost was not editable, documentation was a bit thin on this

This is another known limitation: the fact that `target-postgres`'s

host

cannot be edited in the UI: https://gitlab.com/meltano/meltano/-/issues/2126. The CLI doesn't have this limitation.

2. Setup a docker instance per client, where the client would be the name of the docker container and everything within the image would be preconfigured.

Are you thinking of https://meltano.com/docs/containerization.html? You could definitely have a Meltano project for each client, initialize Git repos, and containerize all of them, but you (and your team) would need to manage those Git repos and Docker images and containers yourself, like you do with any other software engineering projects, without help from a centralized UI.

Wanting to centralise all the connections into a central DB which would become problematic if we are running multiple docker instances per client.

Do you mean that you'd like to load the extracted data for all of your clients into the same DB/data warehouse? That'd work if you configure all of your project's loaders as appropriate. Or are you referring to having all projects store their system databases (https://meltano.com/docs/project.html#system-database) in the same Postgres database? That's not currently possible, but would be once we allow each project to use its own schema in the same DB: https://gitlab.com/meltano/meltano/-/issues/2082

With the docker, is there a variation that would auto-configure a turnkey default project?

Note that if you're containerizing your Meltano project, it will be readonly, so you won't be able to add any new plugins: https://meltano.com/docs/settings.html#project-readonly. You would need to add these to your project before building the project-specific Docker image. Would that work for your team?

Hey mate I guess the ultimate objective is a means of replacing Stitch, currently we setup a new trial account per client which gives us enough bandwidth for basic operations but keen to move to something morein our control. I have attempted the DO droplet and the docker, just unable to get a seamless process to launch an environment, nor being able to get the loader working Looking at Meltano strictly from a UI perspective so its non technical dependent and client friendly. Everything we setup for my clients is built in a way they can take over if they choose too and can easily walk away. Requirements are rather simple, map FB, Google and Salesforce to a data warehouse. Docker Meltano looked like an easier means for the marketing coordinator team whom are not code savvy to manage all the analytic requirements. I just need to resolve a few things Setup process Centralised Destination Databases With docker is there a fully preconfigured default project setup? I am assuming it would be install Meltano then install a default project along with any required config At the moment meltano/meltano:latest is asking to create a project from CLI after launching the docker environment. The questions in regards to Postgres was more for the destinations. The loader documentation is a little confusing as what I have found in the video tutorials doesn't match the latest UI. Still unsure if its locked to a local database or not from the FE UI, if it is then I can't proceed with the strategy I had in mind. So, if we had 5 clients on 5 docker environments, we would want to centralise all the FB stats into a centralised Database separated by users and schemas. So my thinking would be I can setup a client per docker, point each docker to a subdomain. brand1.domain.com, brand2.domain.com etc Centralise all the destinations of all the dockers to 1 centralised database from which the team can then in theory manage all their data belending requirements ie we have clients that have multiple brands with multiple GA/FB/Google Ads accounts, we would setup a centralised DB for the client and a Meltanoo docker per project (brand), point everything at a centralised database for which all other data blending could be managed from.

I have attempted the DO droplet and the docker, just unable to get a seamless process to launch an environment, nor being able to get the loader working

@dwayne Have you tried out the local installation and the CLI by going through the examples on the homepage (https://meltano.com/) or the Getting Started guide (https://meltano.com/docs/getting-started.html)? Even if you'd like to have your clients use the UI, you'd be able to get the loader set up, and it would give you some insight into how Meltano expects you to think about projects, plugins, etc.

Looking at Meltano strictly from a UI perspective so its non technical dependent and client friendly. Everything we setup for my clients is built in a way they can take over if they choose too and can easily walk away.

Like I mentioned, the UI is currently limited in various way compared to the CLI, but the basics (adding plugins to your project, configuring plugins, creating and running pipelines) are there, so it could be a good fit for your clients, although it's currently not nearly as polished as Stitch e.a.

With docker is there a fully preconfigured default project setup? I am assuming it would be install Meltano then install a default project along with any required config

At the moment meltano/meltano:latest is asking to create a project from CLI after launching the docker environment.

The simplest possible Meltano project (which you can create using

meltano init

) is just a directory with a

meltano.yml

file: https://meltano.com/docs/project.html#meltano-yml-project-file. If you have such a directory on your system somewhere, you can mount it into a

meltano/meltano

Docker container and run

meltano ui

(and other CLI commands) from the context of the project. Since plugins and pipeline schedules are stored in

meltano.yml

, your directory needs to be writable and persisted outside of the container, though, so we couldn't ship a Docker image that contains a default project already: the project always needs to be created explicitly as a writable directory on your system, that the Docker container can then use.

The questions in regards to Postgres was more for the destinations.

The loader documentation is a little confusing as what I have found in the video tutorials doesn't match the latest UI. Still unsure if its locked to a local database or not from the FE UI, if it is then I can't proceed with the strategy I had in mind.

`target-postgres`'s

host

can only currently be changed to something other than

localhost

using the CLI or an environment variable, like under https://meltano.com/docs/getting-started.html#configure-the-loader. Editing it in the UI is not currently possible, but that's a bug we intend to fix: https://gitlab.com/meltano/meltano/-/issues/2126. It's not currently prioritized, but that can change if this would be a blocker for you :)

So, if we had 5 clients on 5 docker environments, we would want to centralise all the FB stats into a centralised Database separated by users and schemas.

So my thinking would be I can setup a client per docker, point each docker to a subdomain. brand1.domain.com, brand2.domain.com etc

Centralise all the destinations of all the dockers to 1 centralised database from which the team can then in theory manage all their data belending requirements

ie we have clients that have multiple brands with multiple GA/FB/Google Ads accounts, we would setup a centralised DB for the client and a Meltanoo docker per project (brand), point everything at a centralised database for which all other data blending could be managed from.

That would work, as long as each Docker container running

meltano ui

has access to its own project directory on some kind of persistent storage. Are you planning to use Airflow to run your pipelines on a schedule? Then you'd also need to run `meltano invoke airflow scheduler`: <https://…