Hello @douwe_maan, I would like to know if the project is still going on, I would like to contribute especially to integrate the UI side. So I would like to know: - How does the backend work? - How do the UI APIs work from the controller. Thanks You

@kimoudev Welcome!

I would like to know if the project is still going on,

Definitely! I hope that's clear from the activity in this Slack workspace and the repo at https://gitlab.com/meltano/meltano 😄

I would like to contribute especially to integrate the UI side.

Awesome! We have a bunch of open issues in https://gitlab.com/groups/meltano/-/epics/78 already, and new issues and contributions to address existing ones are more than welcome!

So I would like to know:

- How does the backend work?

What aspect of the backend are you interested in? The core Meltano business logic can be found in the

meltano.core

module (https://gitlab.com/meltano/meltano/-/tree/master/src/meltano/core), the CLI commands that use this core functionality are in

meltano.cli

(https://gitlab.com/meltano/meltano/-/tree/master/src/meltano/cli), and the Flask API that is used by the API can be found at

meltano.api

(https://gitlab.com/meltano/meltano/-/tree/master/src/meltano/api).

- How do the UI APIs work from the controller.

Which controller are you referring to? The controllers in https://gitlab.com/meltano/meltano/-/tree/master/src/meltano/api/controllers define endpoints that the UI can make requests to, like

/plugins/all

defined here: https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/api/controllers/plugins.py#L40

Thank you @douwe_maan  for the precisions. After installation and some testing I saw that all the logic is centralized on the discovery.yml and meltano.yml. Part of the logic is not in the database. My question, meltano wants to keep this architecture or would like to change it in the future? My opinion is that if we want each user to have several identical loaders...etc, the best would be to persist some information in the database. Thanks,

@kimoudev The fact that a Meltano project's main implementation details are stored in

meltano.yml

is a pretty central part of its design, and not something that's likely to change, since it's exactly this feature that brings DevOps best practices such as version control and CI/CD to Meltano projects, as discussed briefly in https://meltano.com/docs/project.html. I shared some more thoughts on storing things in

meltano.yml

instead of the system database in https://gitlab.com/meltano/meltano/-/issues/2306#note_401383373, which you may want to check out if this is a topic of interest to you. Note that plugin configuration can already be stored in the database; see point 3 under https://meltano.com/docs/configuration.html#configuration-layers. If you have multiple users with their own plugins and configurations, your best bet would be to have a separate Meltano project for each user/team/project, so that all implementation details are managed separately and the users and projects aren't in each other's ways. We can definitely explore more scalable ways of managing multiple projects in the UI or the CLI, of course. Please let me know if this helps! If you have a specific use case in mind that doesn't seem to fit the current architecture, I'll gladly discuss it some more.

With a little more analysis, meltano is not built for use in CLOUD type SAAS. Otherwise the questions would be: 1-How can a user have several similar loaders? user1 = {gitlab-tap=>enterprise1, gitlab-tap2=>enterprise2} 2-How to initialize each project according to the user if the server should start on a project. I refer to a similar project in the cloud like www.panoply.io

With a little more analysis, meltano is not built for use in CLOUD type SAAS.

Correct, it's currently optimized for data engineers who want to manage their ELT pipelines like a software engineering project, not through a visual UI. The UI is explicitly lower priority than the CLI right now: https://meltano.com/docs/ui.html

Otherwise the questions would be:

1-How can a user have several similar loaders? user1 = {gitlab-tap=>enterprise1, gitlab-tap2=>enterprise2}

See https://meltano.slack.com/archives/CFG3C3C66/p1602095081147800?thread_ts=1602086759.143100&cid=CFG3C3C66

2-How to initialize each project according to the user if the server should start on a project.

As you've identified, Meltano currently requires projects to be created explicitly using

meltano init

, after which a Git repo can be initialized in the directory which can then be shared and collaborated on using a platform like GitLab or GitHub, to get the benefits of DevOps best practices such as version control and CI/CD. Inside each such project, a project-specific UI can be spun up using

meltano ui

, based on the assumption that a company's data team would have just a single Meltano project all team members can work in. Meltano does not currently have a higher level UI to create and manage projects, although I'll gladly discuss adding such a UI and related CLI tooling to allow Meltano to be used in a multi-tenant SaaS setup. Would you like to create an issue for that?

Yes, I would like to. But time to understand how the meltano.yml file works with TAPS and TARGET and the link "pip_url". I don't have the meltano architecture, I'm trying to understand the logic of the written code but it takes some time Thank you for your help

@kimoudev Did you see https://meltano.com/docs/project.html#plugins and https://meltano.com/docs/plugins.html#extractors?