Hi, I would like ask about the future of

meltano elt

vs

meltano run

. I have heard that it may be possible that

meltano elt

gets deprecated at some stage. I just wanted to talk about a use case where manually setting a --state-id (--job-id in prior versions of meltano) is important to me. It would be a nice feature to have the option to specify --state-id on

meltano run

or the retention of

meltano elt

for when I wish to manage the name of a state id. I have multiple providers supplying me csv files into different s3 buckets (one bucket per provider for security isolation). I wish to run a separate meltano job to extract data from each bucket (using tap-s3-csv) to (target-snowflake). The reason for this is. 1. The loads can take place in parallel, my Airflow framework will kick off all the meltano loads at once for each provider. 2. If one providers files are incorrect structurally and causes a failure, only that providers file load fails, not all other providers meltano loads. For any failed loads, the providers load can be restarted when the files are correct. Currently, I can use the same tap combination i.e. tap-s3-csv and target-snowflake but provide different environment variables to point to the correct bucket. State is import to me because it tracks what files have been loaded and the last modify/create timestamp. For this to work I supply a different --state-id for each providers meltano run. By doing this using the same tap combination, I don't need to create in my meltano.yml file different inherited taps for each provider (which need all the environment variable to be setup). So the control of the --state-id gives me uniqueness for each providers if I use the

elt run

command. So in summary, there are probably different ways to organise my pipeline - I do think having the option to specify your own --state-id is useful, especially when you are using the same pattern but just changing one variable like a bucket name. I hope this makes sense and this considered for the products roadmap. Thanks Steve

A big and important topic - thanks again for raising! Sharing back from our DM: https://github.com/meltano/meltano/issues/6164

I have heard that it may be possible that

meltano elt

gets deprecated at some stage.

Where did you hear this?

Here's the latest and current thinking on the future of `meltano elt`: Future of `meltano elt` or `meltano el` (adapt vs deprecate) · Discussion #6273 · meltano/meltano (github.com) Comments, suggestions, are feedback are all much appreciated.