Meltano

How does state work with custom tap? I am running `CSV_OUTPUT_PATH='./output/zuora' meltano elt tap-zuora target-csv`  and every run I see “No state was found, complete import”. How can I configure Meltano to pass the state from previous run?

something like:
meltano-jdk elt tap-mssql target-postgres --*job_id* tap-mssql

Yep, you’ll want to pass a `--job_id=my-pipeline`
for Meltano to start considering the ELT runs as subsequent runs of the same pipeline

In my first run I did not pass a `--job_id` and I see a job id was auto generated. In my next run I am passing this auto generated job id but still see the message “no state…“.

Is Meltano ignoring state with auto-generated job id?

No, if you reuse the autogenerated one it should work as well. You can look in your system database in the “job” table to see if any singer_state was actually stored in the payload column for that run

The first run would have logged something to indicate state was being stored

In general you’ll want to use your own job IDs though. If you use “meltano schedule” (or the equivalent UI), pipeline names will be used as job IDs and state will thus be handled automatically.

Thanks for the hint <@U06CE9BN604> <https://meltano.slack.com/archives/CFG3C3C66/p1594733588157500?thread_ts=1594732952.145700&amp;cid=CFG3C3C66>

I looked into the system database and found the generated job id is different from the folder name under `.meltano/run/elt/`. Fixing that difference made it work.

Correct Job Id per database: `job_20200714-00:38:49.927082`
Folder name: `job-20200714-003849-927082`

<@U06CE9BN604> It may be nicer to sync these job ids in Meltano to avoid confusion.

<@U06BXJWV7BR> Ah interesting! Those IDs should definitely match. It looks like we generated a string for use in the DB that wasn't actually a valid directory name, resulting in this inconsistency. Wanna create an issue for this bug or shall I? :slightly_smiling_face:

Issue created: <https://gitlab.com/meltano/meltano/-/issues/2167>

Perfect, thanks a lot for creating these!