meltano 2.0 related question about

jobs

- I'm loving the jobs feature and am migrating some projects to 2.0. In order to do what I previously did with

job_id

where I had different invocations of the same

elt

command but with different variations of entities to sync (via use of

--select

parameter in

elt

), how can I do that with

jobs

and the

run

command? Do

extras

work for the

run

command? The

extras

documentation only mentions the

elt

and

invoke

commands? https://docs.meltano.com/concepts/plugins#extras

Hi, @christoph! So glad to hear you are enjoying the jobs feature and 2.0 in general! Re:

In order to do what I previously did with

job_id

where I had different invocations of the same

elt

command but with different variations of entities to sync (via use of

--select

parameter in

elt

), how can I do that with

jobs

and the

run

command?

1. If the distinct state ID (previously called "job ID") was needed to differentiate a different stage or deployment, we recommend using different "environments" for those jobs, just as 'staging' and 'prod', or 'prod_west' and 'prod_east', etc. 2. If the distinct state ID was needed because the same tap was used to sync from two different sources, like

tap-slack-singer

and

tap-slack-meltano

both using the

tap-slack

to pull from two different data sources, then we recommend using 'inherits from' relationships so that the plugins have distinct names when pulling from distinct sources. 3. Assuming your jobs reference distinct source names (point 2 above) and are running in a specific enviornment context (point 1 above), then

meltano run

will autogenerate state IDs for you that will always be unique and correct for that combination of {environment, source, destination}. 4. As a one time migration task, you may want to move/copy your old state artifacts into the new auto-generated naming convention that is used by

meltano run

.

Thanks @aaronsteers! Makes sense! Yes, I was thinking about going down the path in point 2 and create an inherited 'tap' for each of my entities/streams in the tap. I was just a bit concerned about any overhead associated with that approach. Hence I was hoping that I could use the

select

extra in a

run <jobname>

command, and then I would name the jobs myself in the job definitions.

If that works for you use case, that's probably a good path forward. Declaring these select lists as distinct tap instances gives each their own state ID, which has the nice effect of also making them safe to run in parallel, without concern that they could block each other or clobber each other's state on completion.