Hi guys is there any way to disable

system database

? we don’t need it at all it just creates issues in our case

There's no way currently. Run history is still saved in the system database, even when using an external state backend.

$ sqlite3 .meltano/meltano.db ".schema runs"                 
CREATE TABLE IF NOT EXISTS "runs" (
        id INTEGER NOT NULL, 
        job_name VARCHAR, 
        state VARCHAR(7), 
        started_at DATETIME, 
        ended_at DATETIME, 
        payload VARCHAR, 
        payload_flags INTEGER, run_id CHAR(32), "trigger" VARCHAR, last_heartbeat_at DATETIME, 
        PRIMARY KEY (id)
)

Do log a feature request if you can. Also, if you're interested, runs are saved in https://github.com/meltano/meltano/blob/3d04eab02a6b2a26918a98fff935f511581cc230/src/meltano/core/state_service.py#L131-L134

great thanks ! we’re having issues because we have multiple meltano runs in parallel and sqlite can’t handle that moved to postgres but it’s also creating some issues because of amount of parallel executions and since we’re running meltano in short living jobs, all run history in sqlite also got cleared the simpliest solution for now was to use different sqlite db names for each run, so there were no concurrency

Easiest way to avoid other overlapping issues (you'll hit catalog issues as well) is to use a new container for each run and then your db, catalog, etc will be isolated and you won't have any concurrent issues with meltano at least

@visch i wish i could but we have 10k runs not in parallel but we’re batching them, 10 per worker, 200 workers which catalog issues ?

spin up 10k containers? that's what I'd do

our k8s won’t be happy with it 😀

Cloud services work pretty well these days!