Hi all. So i’ve read through all the docs and run through some examples, but there’s one point i’m not really clear on. How do you create multiple pipelines using the same tap in your project? For example, if I wanted to extract from multiple different databases, or from multiple s3 sources?

@andrew_stewart Did you see https://meltano.com/docs/integration.html#pipeline-specific-configuration? There are also plans to support multiple configuration profiles at some point: https://gitlab.com/groups/meltano/-/epics/51

Yeah, I read that but it sounds like that’s just talking about overriding default config values? I assume that could be used for the purpose I described by way of orchestrators (multiple versions of the same DAG using different environment variables).

Yes, that's currently the recommended method: to define configuration values you'll always want for a tap or target in

meltano.yml

, and to then specify pipeline-specific config values (to change the source S3 bucket or target database) at runtime using env vars.

Gotcha

With profiles (see link above), you'd be able to run

meltano elt tap-foo@profile-bar

,

meltano elt tap-foo@profile-baz

, etc, with those profiles each having their own

config

dict in

meltano.yml

, but for the moment env vars are the way to go

Well that will certainly nudge me towards using the orchestration feature.

What are you currently using to manage the automatic running of your

meltano elt

pipelines?

Im curious to dig through that epic to see what thoughts there are regarding multiple profiles.

Ah, sure 🙂 Let me know if I can clarify anything. I expect to get to profiles some time in the next few months

Just to think out loud for benefit of feedback, I think from the perspective of a user used to platforms like Fivetran etc, I was expecting to see some kind of central config table of all my E->L variants, and that ‘meltano elt’ would somehow run through all of those.

With

meltano add orchestrator airflow

,

meltano schedule <pipeline>

,

meltano schedule list

, and

meltano invoke airflow scheduler

that's basically what you get! By the time we direct our focus specifically to less technical users who just want to use the UI, we can abstract that away even more and install and run the orchestrator automatically.

I wonder if multiple profile management could potentially be as simple as a config file that enumerates every tap/target config combination, which could potentially feed into (or inform) DAG generation.

What you're describing is exactly how our Airflow integration works 🙂

As well as potentially then having a variant of

meltano elt -all

or something similar

a config file that enumerates every tap/target config combination, which could potentially feed into (or inform) DAG generation.

That's https://gitlab.com/meltano/files-airflow/-/blob/master/bundle/orchestrate/dags/meltano.py

As well as potentially then having a variant of 

meltano elt -all

 or something similar

Currently, that's

meltano invoke airflow scheduler

, but we could have

meltano orchestrate

: https://gitlab.com/meltano/meltano/-/issues/2231

Ok so two more questions

Some of this orchestration stuff is definitely not as clear from the docs as it should be by the way, which I'll fix eventually: https://gitlab.com/meltano/meltano/-/issues/2260

1) where in the project can I currently represent those multiple profiles? Would this basically involve making copies of the DAG?

But with our current focus on data engineers who are already comfortable with orchestration and tools like Airflow, addressing that not been a priority yet

2) you just answered with issue 2260

meltano schedule

schedules can also be managed using

meltano ui

Pipelines tab, by the way, but the UI does not automatically run the orchestrator

Thanks for your patience with my newbie questions. I’m stoked to try this :)

So sounds like all the ingredients are there, I just gotta better familiarize myself with the interoperations.

And I've got to improve the docs 🙂 But yes, the ingredients are here, and it looks remarkably close to what you were brainstorming already!

I didnt really look yet, but are there any example repos setup that demonstrate more complicated configs?

Thanks for your patience with my newbie questions. I’m stoked to try this 🙂

Thanks for your feedback! It's really useful to hear what is and isn't obvious to newbies 😄

as in beyond just a single tap/target

ie a ‘real world’ project

https://gitlab.com/meltano/demo-project is not real, but it does have 2 schedules: https://gitlab.com/meltano/demo-project/-/blob/master/meltano.yml#L71

Gotcha. I think what’s been hanging me up here is that

meltano elt

seems to both 1) register a pipeline and 2) run the pipeline (initial run).

@andrew_stewart The way I think about it, it runs a pipeline (whether it's the first or the last run, it doesn't matter, you'll always use

meltano elt

), and it allows you to specify a job ID to tag the state with. I get why you're interpreting that tagging as "registering the pipeline", though, since (unless you're using the default timestamped job ID) you are naming the pipeline for later reference. The pipeline won't exist in

meltano.yml

or

meltano schedule list

or Airflow until it's explicitly registered along with its arguments using

meltano schedule

, though, since running a pipeline once using

meltano elt

doesn't necessarily mean you want to run it again, even if a job ID was provided. Do you think this could more intuitive?

Maybe, but it’s also possible I’m being overly influenced by

dbt

where you configure/define the elements of your project and then

dbt run

the whole project.

The difference is that to Meltano, running

meltano elt

shouldn't be considered "running the project" or "defining a pipeline", but rather just "running a pipeline" as a one-off action without impact on the project. The fact that it "registers" the job ID should not be considered "defining an element of the project", since the state and job ID are stored in the system database, which is not considered part of the project since it's environment-specific and not version controlled. Actually defining the pipeline as an element of your project is done using

meltano schedule <name> ...

, which modifies

meltano.yml

and would usually be followed by a Git commit, unlike

meltano elt

, which doesn't touch the project at all.

Is a project intended to encapsulate a single extractor/loader combo ?

Of course, you don't need to register the pipeline as an element of your project to use Meltano, and it's perfectly valid to not use

meltano schedule

at all and just call

meltano elt

with the appropriate arguments directly from

BashOperator

(or its non-Airflow equivalent). For users already using Airflow or another orchestrator, treating

meltano elt

as just a CLI they can use to run Singer taps and targets may be more appropriate than to buy into Meltano's idea of explicitly defined pipelines that are expected to picked up by a DAG generator that reads from

meltano schedule list --format=json

, which is why

meltano elt

supports a job ID ("extractor state tag") that doesn't need to have been explicitly registered with the project before.

what do you think configuration profiles will look like ?

@andrew_stewart This part is already implemented actually, it's just not yet taken into account across the board, which that epic is meant to address. In

meltano.yml

, every plugin can have a

profiles

array, where each profile has a

name

, and its own

config

object:

extractors:
  - name: tap-gitlab
    pip_url: git+<https://gitlab.com/meltano/tap-gitlab.git>
    config:
      projects: meltano/meltano
      start_date: '2020-05-01'
    profiles:
      - name: demo-project
        config:
          projects: meltano/demo-project

On

meltano config

,

meltano elt

, etc, you'd reference that specific profile as

tap-gitlab@demo-project

, e.g.

meltano elt tap-gitlab@demo-project target-snowflake@foo ...

ahaaaaa

The idea is that profile config inherits from the base (default) config, so the combined config for

tap-gitlab@demo-project

would be

{ projects: meltano/demo-project, start_date: '2020-05-01' }

Sure, that makes total sense.

I'm looking forward to being able to rewrite https://meltano.com/docs/integration.html#pipeline-specific-configuration with that feature in mind 😄

Right!

I wouldn't be opposed to supporting an

--all

flag on the proposed

meltano schedule run

: https://gitlab.com/meltano/meltano/-/issues/2227

Right.. it’s funny because as I think about all the pieces together, it wouldn’t really make sense, but it sorta makes sense from a tutorial / sandbox purpose.

Right. Would you expect

meltano schedule run --all

to run pipelines in parallel, or serially? What if one pipeline fails? What would it log? (Especially if running in parallel!)

for a local run? probably just serially, and abort on fail.

Which is not to say that there isn't value in that 🙂 But it may also give people the wrong idea of how pipelines are usually run "in the real world", where serially and aborting on the first failure makes little sense

Yup. Well you’ve certainly made a very complex problem much simpler than it was.

Any feedback on how to make it simpler is appreciated of course 🙂 But some things are necessarily complex!

Thinking about it a bit more.. if there was really demand for some kind of single “task-naive” entrypoint (ike

dbt run

), I suppose what you could actually do is eventually have a

meltano run

command that fires up the docker-compose complete with the Meltano UI, orchestrator of choice, etc etc etc. Basically a ‘main switch’.